-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move periodic job to ES repo #48570
Move periodic job to ES repo #48570
Changes from 1 commit
3fe2504
4d0b7eb
86049af
811b707
cc042d1
117cbd6
b6342cc
f925c5b
2d52f58
0da414f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
#!/bin/bash | ||
|
||
# drop page cache and kernel slab objects on linux | ||
[[ -x /usr/local/sbin/drop-caches ]] && sudo /usr/local/sbin/drop-caches | ||
|
||
rm -Rfv ~/.gradle/init.d/init.gradle | ||
mkdir -p ~/.gradle/init.d && cp -v $WORKSPACE/.ci/init.gradle ~/.gradle/init.d | ||
|
||
if [ -f /proc/cpuinfo ] ; then | ||
MAX_WORKERS=`grep '^cpu\scores' /proc/cpuinfo | uniq | sed 's/\s\+//g' | cut -d':' -f 2` | ||
else | ||
if [[ "$OSTYPE" == "darwin"* ]]; then | ||
MAX_WORKERS=`sysctl -n hw.physicalcpu | sed 's/\s\+//g'` | ||
# Looks like it's too much for our workers so reduce it further | ||
MAX_WORKERS=$(($MAX_WORKERS/2)) | ||
else | ||
echo "Unsupported OS Type: $OSTYPE" | ||
exit 1 | ||
fi | ||
fi | ||
|
||
if pwd | grep -v -q ^/dev/shm ; then | ||
echo "Not running on a ramdisk, reducing number of workers" | ||
MAX_WORKERS=$(($MAX_WORKERS*2/3)) | ||
fi | ||
|
||
export GRADLE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -Xmx128m -Xms128m" | ||
set -e | ||
./gradlew --parallel --scan \ | ||
-Dorg.elasticsearch.build.cache.url=https://gradle-enterprise.elastic.co/cache/ \ | ||
--parallel --max-workers=$MAX_WORKERS \ | ||
"$@" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
--- | ||
|
||
##### GLOBAL METADATA | ||
|
||
- meta: | ||
cluster: elasticsearch-ci | ||
|
||
##### JOB DEFAULTS | ||
|
||
- job: | ||
vault: | ||
url: https://secrets.elastic.co:8200 | ||
role_id: 1ba1ac3e-aee4-d040-d9a3-6ae23bd2b3db | ||
node: "general-purpose" | ||
concurrent: true | ||
logrotate: | ||
daysToKeep: 30 | ||
numToKeep: 90 | ||
artifactDaysToKeep: 7 | ||
scm: | ||
- git: | ||
name: origin | ||
# master node jenkins user ~/.ssh | ||
credentials-id: f6c7695a-671e-4f4f-a331-acdce44ff9ba | ||
reference-repo: "/var/lib/jenkins/.git-references/elasticsearch.git" | ||
branches: | ||
- "%BRANCH%" | ||
url: "https://github.com/atorok/elasticsearch.git" | ||
basedir: "" | ||
wipe-workspace: "yes" | ||
wrappers: | ||
- timeout: | ||
type: absolute | ||
timeout: "120" | ||
fail: true | ||
- ansicolor | ||
- timestamps | ||
# TODO: No support un JJBB ? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We had to use a custom JJB module for this one. I'm not sure how that translate to JJBB. We need to sort this out though as this is how we gather published build scans from Jenkins. Without this HOMER won't be able to find build scans either so this is a show stopper IMO unless we can sort it out. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was planing on sorting this out before I remove the job from the .ci repo. |
||
# - gradle-build-scan | ||
properties: | ||
- github: | ||
url: https://github.com/elastic/elasticsearch/ | ||
- inject: | ||
properties-content: | | ||
HOME=$JENKINS_HOME | ||
builders: | ||
- inject: | ||
properties-file: '.ci/java-versions.properties' | ||
properties-content: | | ||
JAVA_HOME=$HOME/.java/$ES_BUILD_JAVA | ||
RUNTIME_JAVA_HOME=$HOME/.java/$ES_RUNTIME_JAVA | ||
JAVA7_HOME=$HOME/.java/java7 | ||
JAVA8_HOME=$HOME/.java/java8 | ||
JAVA9_HOME=$HOME/.java/java9 | ||
JAVA10_HOME=$HOME/.java/java10 | ||
JAVA11_HOME=$HOME/.java/java11 | ||
JAVA12_HOME=$HOME/.java/openjdk12 | ||
JAVA13_HOME=$HOME/.java/openjdk13 | ||
- shell: | | ||
#!/usr/local/bin/runbld --redirect-stderr {runbld_args} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does JJB support parameterization? What does There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. you are correct, I changed it to default to last good commit. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 I think if the expectation is that the |
||
.ci/build.sh check | ||
publishers: | ||
- email: | ||
recipients: infra-root+build@elastic.co | ||
# Upload additional logs | ||
- google-cloud-storage: | ||
credentials-id: 'elasticsearch-ci-gcs-plugin' | ||
uploads: | ||
- classic: | ||
file-pattern: 'build/*.tar.bz2' | ||
storage-location: 'gs://elasticsearch-ci-artifacts/jobs/$JOB_NAME' | ||
share-publicly: false | ||
upload-for-failed-jobs: true | ||
show-inline: true | ||
# Notify homer | ||
- postbuildscript: | ||
builders: | ||
- role: SLAVE | ||
build-on: | ||
- SUCCESS | ||
- FAILURE | ||
- UNSTABLE | ||
build-steps: | ||
- shell: | | ||
curl -sS -X POST \ | ||
-H "Content-Type: text/plain" \ | ||
--data "$BUILD_URL" \ | ||
"https://homer.app.elstc.co/webhook/jenkins/buil | ||
mark-vieira marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
- job: | ||
name: elastic+elasticsearch+%BRANCH%+periodic-next | ||
workspace: /dev/shm/elastic+elasticsearch+%BRANCH%+periodic | ||
display-name: "elastic / elasticsearch # %BRANCH% - periodic (experimental)" | ||
description: "Periodic testing of the Elasticsearch %BRANCH% branch.\n" | ||
triggers: | ||
- timed: "H H/1 * * *" | ||
builders: | ||
- shell: | | ||
#!/usr/local/bin/runbld --redirect-stderr --last-good-commit elastic+elasticsearch+{branch}+git+push | ||
mark-vieira marked this conversation as resolved.
Show resolved
Hide resolved
|
||
.ci/build.sh -Dbwc.checkout.align=true check |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
#!/bin/bash | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps we should have a .ci/scripts directory? Seems we are accumulating a lot of different shell scripts all at the root. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's a good idea. I'll move that in the next iteration, when I have to re-test this anyhow. |
||
|
||
if [ -z "$BRANCH" ] ; then | ||
echo "BRANCH is unset" | ||
exit 1 | ||
fi | ||
|
||
rm -Rf .ci/jobs | ||
cp -r .ci/jobs.t .ci/jobs | ||
|
||
sed -i "s/%BRANCH%/${BRANCH}/g" .ci/jobs/*.yml |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be the main repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch!