Ant target to switch between different Solr servers on local env

With Solr 8’s support ending in June 2024, it’s crucial for local environments to transition to Solr 9. Unlike cloud installations of SAP Commerce (where you can easily switch Solr versions via manifest.json), local setups encounter difficulties in switching between Solr versions.

Problem Description

OOTB, SAP Commerce allows switching between Solr server versions using the solrserver.solr.server.version property. However, during the solrserver_after_build callback, the build fails with an “Unable to resolve artifact” exception. This occurs because the OOTB implementation downloads the specified Solr server from the Maven repository, which was last updated in 2015, featuring the latest available Solr server version 4.10.4.

Technical Solution

Certainly, it is possible to configure SAP Commerce to use a standalone version. However, this would require manually copying SAP Commerce customizations into the manually downloaded Solr server distribution, copying project-specific Solr configurations, and ensuring the definition of environment variables for keystore and HTTPS setup before starting Solr itself.

Such an approach appears to be complicated and time-consuming. A better alternative would involve introducing a separate ant target that downloads Solr servers and leverages the existing OOTB infrastructure from the solrserver extension.

Implementation details

The solution centers around the custom ant target named installAllMissingSolrServers, and its functionality is detailed below in a step-by-step manner.

Detection of Desired Solr Server Version:
- The process begins by detecting the desired Solr server version specified in the solrserver.solr.server.version property.
Iteration Over Available Solr Customization Versions:
- The target iterates through the list of available Solr customization versions specified in the solrserver.solr.customizations.versions property.
Verification of Solr Server Installation:
- For each customization version, the target verifies whether the corresponding Solr server is installed and matches the selected Solr server version.
Resolution of Download URL and Version:
- If the Solr server is not installed, the target resolves the download URL and the latest available Solr server version from the Apache distribution site using a custom macrodef resolveSolrDistributionDownloadUrl.
- In the custom macrodef searchSolrDistributionInApacheArchive, the resolution process checks both https://archive.apache.org/dist/solr/solr and https://archive.apache.org/dist/lucene/solr.
- In the custom macrodef resolveSolrVersionForDownload, the latest available patch version or the version specified in the solrserver.solr.server.version property is determined. However, the “after_build” macrodef callback executes the installSolr macrodef, which checks whether solrserver.solr.server.version is installed and removes all other versions. Therefore, a regular build fails, and we can’t always download the latest available version. For example, we can’t download the latest available Solr server 8.11.3 for Solr 8.11, as the regular build would fail due to the OOTB value of solrserver.solr.server.version being 8.11.2.
Download and Installation of Solr Server:
- Upon obtaining a valid download URL, the target proceeds to download the distribution into the /tmp folder.
- The downloaded Solr server distribution is then extracted in the solrserver extension and is configured using the OOTB configureSolrServer macrodef.
Verification of Successful Installation:
- Post-installation, the target verifies whether the selected Solr server version is successfully installed.
- If the desired Solr server version is not installed, the build process fails. This intentional failure is designed to alert developers and guide them in updating the solrserver.solr.server.version property with an existing Solr server version.
- Failing the build serves as a proactive measure to highlight that regular builds will also fail due to the installSolr macrodef OOTB behavior.

Implementation for ant installAllMissingSolrServers

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<project name="blog_solr">

    <target name="installAllMissingSolrServers">
        <property name="selectedSolrServerVersion" value="${solrserver.solr.server.version}"/>
        <echo message="Solr server selected for usage: ${selectedSolrServerVersion}"/>

        <var name="existsSelectedSolrServer" value="false"/>

        <for list="${solrserver.solr.customizations.versions}" param="solrCustomizationVersion">
            <sequential>
                <local name="solrServerVersion"/>
                <solrVersionPathProperty name="solrServerVersion" version="@{solrCustomizationVersion}"/>
                <local name="solrServerPath"/>
                <property name="solrServerPath"
                          value="${ext.solrserver.path}/resources/solr/${solrServerVersion}/server"/>
                <local name="solrCustomizationsPath"/>
                <property name="solrCustomizationsPath"
                          value="${ext.solrserver.path}/resources/solr/${solrServerVersion}/customizations"/>

                <local name="meta.version"/>
                <local name="meta.customizationsVersion"/>
                <property file="${solrServerPath}/meta.properties" prefix="meta."/>

                <property name="meta.version" value=""/>
                <property name="meta.customizationsVersion" value=""/>

                <echo message="--- Solr server ${solrServerVersion} ---"/>
                <echo message="currentVersion: ${meta.version}"/>
                <echo message="expectedCustomizationsVersion: @{solrCustomizationVersion}"/>
                <echo message="currentCustomizationsVersion: ${meta.customizationsVersion}"/>

                <if>
                    <equals arg1="${meta.version}" arg2="${selectedSolrServerVersion}" trim="true"/>
                    <then>
                        <var name="existsSelectedSolrServer" value="true"/>
                    </then>
                </if>

                <if>
                    <not>
                        <and>
                            <available file="${solrServerPath}" type="dir"/>
                            <contains string="${meta.version}" substring="${solrServerVersion}"/>
                            <equals arg1="@{solrCustomizationVersion}" arg2="${meta.customizationsVersion}"
                                    trim="true"/>
                        </and>
                    </not>
                    <then>
                        <echo message="Installing Solr server [version=${solrServerVersion}, customizationsVersion=@{solrCustomizationVersion}]"/>

                        <local name="solrServerDownloadUrl"/>
                        <local name="resolvedSolrServerVersion"/>
                        <resolveSolrDistributionDownloadUrl
                                downloadUrlReturnProperty="solrServerDownloadUrl"
                                resolvedSolrServerVersionReturnProperty="resolvedSolrServerVersion"/>

                        <if>
                            <length string="${solrServerDownloadUrl}" trim="true" when="greater" length="0"/>
                            <then>
                                <echo message="Downloading can take few minutes. Please, wait patiently."/>
                                <get src="${solrServerDownloadUrl}"
                                     dest="${java.io.tmpdir}/solr-${resolvedSolrServerVersion}.tgz"
                                     verbose="false"
                                     usetimestamp="true"/>

                                <delete dir="${solrServerPath}"/>
                                <untar src="${java.io.tmpdir}/solr-${resolvedSolrServerVersion}.tgz"
                                       dest="${solrServerPath}"
                                       compression="gzip">
                                    <cutdirsmapper dirs="1"/>
                                </untar>

                                <configureSolrServer solrServerPath="${solrServerPath}"
                                                     solrCustomizationsPath="${solrCustomizationsPath}"/>

                                <propertyfile file="${solrServerPath}/meta.properties">
                                    <entry key="version" value="${resolvedSolrServerVersion}"/>
                                    <entry key="customizationsVersion" value="@{solrCustomizationVersion}"/>
                                    <entry key="solrServerDistributionUrl" value="${solrServerDownloadUrl}"/>
                                </propertyfile>

                                <echo message="Solr server ${resolvedSolrServerVersion} installation finished."/>

                                <if>
                                    <equals arg1="${resolvedSolrServerVersion}" arg2="${selectedSolrServerVersion}"
                                            trim="true"/>
                                    <then>
                                        <var name="existsSelectedSolrServer" value="true"/>
                                    </then>
                                </if>
                            </then>
                        </if>

                    </then>
                </if>
            </sequential>
        </for>

        <echo message="---------------------------------------------------------------"/>
        <echo message="Switch solr server versions using property solrserver.solr.server.version. Use solr server version as value, not customization version."/>
        <echo message="Keep in mind that after ant build unused solr servers would be cleaned up."/>
        <echo message="---------------------------------------------------------------"/>
        <if>
            <not>
                <equals arg1="${existsSelectedSolrServer}" arg2="true"/>
            </not>
            <then>
                <fail message="Selected solr server ${selectedSolrServerVersion} were not installed. Update solrserver.solr.server.version property with existing solr server version."/>
            </then>
        </if>
    </target>

    <macrodef name="resolveSolrDistributionDownloadUrl">
        <attribute name="downloadUrlReturnProperty"/>
        <attribute name="resolvedSolrServerVersionReturnProperty"/>
        <sequential>
            <var name="@{downloadUrlReturnProperty}" unset="true"/>
            <var name="@{resolvedSolrServerVersionReturnProperty}" unset="true"/>

            <local name="solrArchiveDownloadUrl"/>
            <local name="solrArchiveResolvedSolrServerVersion"/>
            <searchSolrDistributionInApacheArchive solrServerVersion="${solrServerVersion}"
                                                   apacheArchiveUrl="https://archive.apache.org/dist/solr/solr"
                                                   downloadUrlReturnProperty="solrArchiveDownloadUrl"
                                                   resolvedSolrServerVersionReturnProperty="solrArchiveResolvedSolrServerVersion"/>

            <if>
                <length string="${solrArchiveDownloadUrl}" trim="true" when="greater" length="0"/>
                <then>
                    <property name="@{downloadUrlReturnProperty}" value="${solrArchiveDownloadUrl}"/>
                    <property name="@{resolvedSolrServerVersionReturnProperty}"
                              value="${solrArchiveResolvedSolrServerVersion}"/>
                </then>
                <else>
                    <local name="luceneArchiveDownloadUrl"/>
                    <local name="luceneResolvedSolrServerVersion"/>
                    <searchSolrDistributionInApacheArchive solrServerVersion="${solrServerVersion}"
                                                           apacheArchiveUrl="https://archive.apache.org/dist/lucene/solr"
                                                           downloadUrlReturnProperty="luceneArchiveDownloadUrl"
                                                           resolvedSolrServerVersionReturnProperty="luceneResolvedSolrServerVersion"/>

                    <property name="@{downloadUrlReturnProperty}" value="${luceneArchiveDownloadUrl}"/>
                    <property name="@{resolvedSolrServerVersionReturnProperty}"
                              value="${luceneResolvedSolrServerVersion}"/>
                </else>
            </if>
        </sequential>
    </macrodef>

    <macrodef name="searchSolrDistributionInApacheArchive">
        <attribute name="solrServerVersion"/>
        <attribute name="apacheArchiveUrl"/>
        <attribute name="downloadUrlReturnProperty"/>
        <attribute name="resolvedSolrServerVersionReturnProperty"/>
        <sequential>
            <var name="@{downloadUrlReturnProperty}" unset="true"/>
            <var name="@{resolvedSolrServerVersionReturnProperty}" unset="true"/>

            <echo message="Searching Solr distribution @{solrServerVersion} on @{apacheArchiveUrl}"/>
            <get src="@{apacheArchiveUrl}/"
                 dest="${java.io.tmpdir}/apacheArchive.html"
                 quiet="true"/>

            <local name="availableSolrServerVersions"/>
            <loadfile srcFile="${java.io.tmpdir}/apacheArchive.html"
                      property="availableSolrServerVersions">
                <filterchain>
                    <linecontains>
                        <contains value="@{solrServerVersion}"/>
                    </linecontains>
                    <tokenfilter delimoutput=",">
                        <replaceregex pattern=".*&lt;a.*href=[&quot;']?([^&gt;&quot;']*).*&gt;[^&lt;]*"
                                      replace="\1" flags="gi"/>
                        <replacestring from="/" to=""/>
                        <trim/>
                        <ignoreblank/>
                        <uniqfilter/>
                    </tokenfilter>
                </filterchain>
            </loadfile>

            <local name="solrServerVersionForDownload"/>
            <resolveSolrVersionForDownload solrServerVersions="${availableSolrServerVersions}"
                                           returnProperty="solrServerVersionForDownload"/>

            <if>
                <and>
                    <length string="${solrServerVersionForDownload}" trim="true" when="greater" length="0"/>
                </and>
                <then>
                    <echo message="Found Solr server: ${solrServerVersionForDownload}"/>

                    <property name="@{downloadUrlReturnProperty}"
                              value="@{apacheArchiveUrl}/${solrServerVersionForDownload}/solr-${solrServerVersionForDownload}.tgz"/>
                    <property name="@{resolvedSolrServerVersionReturnProperty}"
                              value="${solrServerVersionForDownload}"/>
                </then>
                <else>
                    <echo message="Solr server were not found on @{apacheArchiveUrl}"/>
                    <property name="@{downloadUrlReturnProperty}" value=""/>
                    <property name="@{resolvedSolrServerVersionReturnProperty}" value=""/>
                </else>
            </if>
        </sequential>
    </macrodef>

    <macrodef name="resolveSolrVersionForDownload">
        <attribute name="solrServerVersions"/>
        <attribute name="returnProperty"/>
        <sequential>
            <var name="@{returnProperty}" unset="true"/>

            <var name="useUserSelectedSolrVersion" value="false"/>
            <for list="@{solrServerVersions}" param="version" delimiter=",">
                <sequential>
                    <if>
                        <equals arg1="@{version}" arg2="${selectedSolrServerVersion}"/>
                        <then>
                            <var name="useUserSelectedSolrVersion" value="true"/>
                        </then>
                    </if>

                    <var name="latestAvailableVersion" value="@{version}"/>
                </sequential>
            </for>

            <if>
                <equals arg1="${useUserSelectedSolrVersion}" arg2="true"/>
                <then>
                    <property name="@{returnProperty}" value="${selectedSolrServerVersion}"/>

                    <if>
                        <not>
                            <equals arg1="${latestAvailableVersion}" arg2="${selectedSolrServerVersion}"/>
                        </not>
                        <then>
                            <echo message="Would be used solr version ${selectedSolrServerVersion} instead of latest available version ${latestAvailableVersion}."/>
                        </then>
                    </if>
                </then>
                <else>
                    <if>
                        <not>
                            <contains string="${latestAvailableVersion}" substring="{"/>
                        </not>
                        <then>
                            <property name="@{returnProperty}" value="${latestAvailableVersion}"/>
                        </then>
                        <else>
                            <property name="@{returnProperty}" value=""/>
                        </else>
                    </if>
                </else>
            </if>

        </sequential>
    </macrodef>

</project>

Some ant tricks used for implementation

The task checks for the presence of “{” in the ${latestAvailableVersion} property to check if property were set or no. Ant resolves not set property into a string with the name of the property itself, such as “${propertyName}”. Thus, by checking for the presence of “{”, the script ensures that the property is indeed set to a specific value and not left unset or unresolved.
Empty value is explicitly assigned into return property to use a simple 0 length check in subsequent if tasks to determine if property were not set.
The task is used to define local properties within the scope of the task. This ensures that the properties are not persisted globally, avoiding conflicts in other parts of the build script.

How to use on local environment

In local.properties define property solrserver.solr.server.version to set up desired version of Solr server. For example, 9.2.1 for Solr 9 and SAP Commerce 2211 Patch 19.
Run ant installAllMissingSolrServers to download supported Solr servers and install them in solrserver extension
Run ant installSolr to ensure that OOTB ant build would not fail
Startup Hybris

In case of any issues with Solr startup - remove indexed data by cleaning up folder core-customize/hybris/data/solr/instances/default/. After successful startup reindex all indexes.

Appendix

Example of build failure with usage of custom solr server defined in via solrserver.solr.server.version property:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73


[echo] Installing Solr server [version=9.2.0, type=zip, customizationsVersion=9.2.12]
[artifact:dependencies] Downloading: org/apache/solr/solr/9.2.0/solr-9.2.0.pom from repository central at https://repo1.maven.org/maven2
[artifact:dependencies] Unable to locate resource in repository
[artifact:dependencies] [INFO] Unable to find resource 'org.apache.solr:solr:pom:9.2.0' in repository central (https://repo1.maven.org/maven2)
[artifact:dependencies] Downloading: org/apache/solr/solr/9.2.0/solr-9.2.0.zip from repository central at https://repo1.maven.org/maven2
[artifact:dependencies] Unable to locate resource in repository
[artifact:dependencies] [INFO] Unable to find resource 'org.apache.solr:solr:zip:9.2.0' in repository central (https://repo1.maven.org/maven2)
[artifact:dependencies] An error has occurred while processing the Maven artifact tasks.
[artifact:dependencies]  Diagnosis:
[artifact:dependencies] 
[artifact:dependencies] Unable to resolve artifact: Missing:
[artifact:dependencies] ----------
[artifact:dependencies] 1) org.apache.solr:solr:zip:9.2.0
[artifact:dependencies] 
[artifact:dependencies]   Try downloading the file manually from the project website.
[artifact:dependencies] 
[artifact:dependencies]   Then, install it using the command: 
[artifact:dependencies]       mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr -Dversion=9.2.0 -Dpackaging=zip -Dfile=/path/to/file
[artifact:dependencies] 
[artifact:dependencies]   Alternatively, if you host your own repository you can deploy the file there: 
[artifact:dependencies]       mvn deploy:deploy-file -DgroupId=org.apache.solr -DartifactId=solr -Dversion=9.2.0 -Dpackaging=zip -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[artifact:dependencies] 
[artifact:dependencies]   Path to dependency: 
[artifact:dependencies]   	1) org.apache.maven:super-pom:pom:2.0
[artifact:dependencies]   	2) org.apache.solr:solr:zip:9.2.0
[artifact:dependencies] 
[artifact:dependencies] ----------
[artifact:dependencies] 1 required artifact is missing.
[artifact:dependencies] 
[artifact:dependencies] for artifact: 
[artifact:dependencies]   org.apache.maven:super-pom:pom:2.0
[artifact:dependencies] 
[artifact:dependencies] from the specified remote repositories:
[artifact:dependencies]   central (https://repo1.maven.org/maven2)
[artifact:dependencies] 
[artifact:dependencies] 

BUILD FAILED
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/build.xml:13: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/compiling.xml:95: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/compiling.xml:136: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/util.xml:42: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/util.xml:44: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/compiling.xml:143: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/compiling.xml:404: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/platform/resources/ant/util.xml:142: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/modules/search-and-navigation/solrserver/buildcallbacks.xml:164: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/modules/search-and-navigation/solrserver/buildcallbacks.xml:210: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/modules/search-and-navigation/solrserver/buildcallbacks.xml:311: The following error occurred while executing this line:
/home/clutcher/projects/blog/core-customize/hybris/bin/modules/search-and-navigation/solrserver/buildcallbacks.xml:413: Unable to resolve artifact: Missing:
----------
1) org.apache.solr:solr:zip:9.2.0

  Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=org.apache.solr -DartifactId=solr -Dversion=9.2.0 -Dpackaging=zip -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=org.apache.solr -DartifactId=solr -Dversion=9.2.0 -Dpackaging=zip -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

  Path to dependency: 
  	1) org.apache.maven:super-pom:pom:2.0
  	2) org.apache.solr:solr:zip:9.2.0

----------
1 required artifact is missing.

for artifact: 
  org.apache.maven:super-pom:pom:2.0

from the specified remote repositories:
  central (https://repo1.maven.org/maven2)