Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Build bzip2 from source for Linux Pyinstaller builds #6924

Merged
merged 7 commits into from
Apr 10, 2024
Merged

chore: Build bzip2 from source for Linux Pyinstaller builds #6924

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0d2a3df
Build bzip2 from source
lucashuy Apr 9, 2024
1c247dc
Build shared object
lucashuy Apr 9, 2024
bb9c5f0
Add bzip shared library to output
lucashuy Apr 9, 2024
439988b
Symlinked bzip and make run in parallel
lucashuy Apr 9, 2024
9e6a241
Add libcrypto back to spec file and fix indenting
lucashuy Apr 9, 2024
a7e9459
Added comment explaining install_sw
lucashuy Apr 9, 2024
7ee4c30
Remove including shared library to use the one's found on the system
lucashuy Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 39 additions & 10 deletions installer/pyinstaller/build-linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ python_library_zip_filename=$2
build_binary_name=$3
build_folder=$4
python_version=$5
openssl_version=$6
zlib_version=$7

if [ "$python_library_zip_filename" = "" ]; then
python_library_zip_filename="python-libraries.zip";
Expand All @@ -13,6 +15,14 @@ if [ "$python_version" = "" ]; then
python_version="3.11.8";
fi

if [ "$openssl_version" = "" ]; then
openssl_version="1.1.1w";
fi

if [ "$zlib_version" = "" ]; then
zlib_version="1.3.1";
fi

if [ "$CI_OVERRIDE" = "1" ]; then
build_folder="aws-sam-cli-beta"
build_binary_name="sam-beta"
Expand All @@ -28,7 +38,7 @@ fi

set -eux

yum install -y libffi-devel bzip2-devel
yum install -y libffi-devel

echo "Making Folders"
mkdir -p .build/src
Expand All @@ -39,17 +49,36 @@ mkdir -p .build/output/openssl
cd .build/output/openssl

echo "Building OpenSSL"
curl "https://www.openssl.org/source/openssl-1.1.1w.tar.gz" --output openssl-1.1.1.tar.gz
tar xzf openssl-1.1.1.tar.gz
cd openssl-1.1.1w
./config --prefix=/opt/openssl && make -j8 && make install
cd ../../..
curl "https://www.openssl.org/source/openssl-${openssl_version}.tar.gz" --output openssl.tar.gz
tar xzf openssl.tar.gz
cd openssl-${openssl_version}
./config --prefix=/opt/openssl && make -j8 && make -j8 install_sw
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The install_sw target installs OpenSSL without all the manual pages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, let me just add a comment for this in the scripts

cd ../../

echo "Building zlib"
curl https://www.zlib.net/zlib-1.3.1.tar.gz --output zlib.tar.gz
curl https://www.zlib.net/zlib-${zlib_version}.tar.gz --output zlib.tar.gz
tar xvf zlib.tar.gz
cd zlib-1.3.1
./configure && make -j8 && make install
cd zlib-${zlib_version}
./configure && make -j8 && make -j8 install
cd ../

echo "Building bzip2"
mkdir bzip2 && cd bzip2
git init
git remote add origin https://gitlab.com/bzip2/bzip2.git
# this is the 1.0.8 release
# https://gitlab.com/bzip2/bzip2/-/tags
# fetch specific commit as to not grab the entire git history
git fetch origin 6a8690fc8d26c815e798c588f796eabe9d684cf0
git reset --hard FETCH_HEAD
make -j8 -f Makefile-libbz2_so
cp libbz2.so.1.0.8 /usr/local/lib
ln -s /usr/local/lib/libbz2.so.1.0.8 /usr/local/lib/libbz2.so.1.0
ln -s /usr/local/lib/libbz2.so.1.0 /usr/local/lib/libbz2.so.1
make -j8 install
cd ../

# Return to `.build/` folder
cd ../

echo "Copying Source"
Expand Down Expand Up @@ -81,7 +110,7 @@ cd Python-$python_version
--with-openssl=/opt/openssl \
--with-openssl-rpath=auto
make -j8
make install
make -j8 install
ldconfig
cd ..

Expand Down
6 changes: 3 additions & 3 deletions installer/pyinstaller/build-mac.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ cd openssl-"$openssl_version"
# Openssl configure https://wiki.openssl.org/index.php/Compilation_and_Installation
./Configure --prefix=/usr/local --openssldir=/usr/local/openssl no-ssl3 no-ssl3-method no-zlib ${openssl_config_arch} enable-ec_nistp_64_gcc_128

make
sudo make install
make -j8
sudo make -j8 install_sw
cd ..

# Copying aws-sam-cli source code
Expand Down Expand Up @@ -93,7 +93,7 @@ tar -xzf python.tgz
cd Python-"$python_version"
./configure --enable-shared
make -j8
sudo make install
sudo make -j8 install
cd ..

echo "Installing Python Libraries"
Expand Down
56 changes: 31 additions & 25 deletions installer/pyinstaller/samcli-mac.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,35 @@
# -*- mode: python -*-
block_cipher = None
exe_name = 'sam'
analysis = Analysis(['../../samcli/__main__.py'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=['./installer/pyinstaller'],
runtime_hooks=[],
excludes=[],
cipher=block_cipher)
analysis = Analysis(
['../../samcli/__main__.py'],
binaries=[],
datas=[],
hiddenimports=[],
hookspath=['./installer/pyinstaller'],
runtime_hooks=[],
excludes=[],
cipher=block_cipher
)
pyz = PYZ(analysis.pure, analysis.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
analysis.scripts,
[],
exclude_binaries=True,
name=exe_name,
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
analysis.binaries,
analysis.zipfiles,
analysis.datas,
strip=False,
upx=True,
name='sam')
exe = EXE(
pyz,
analysis.scripts,
[],
exclude_binaries=True,
name=exe_name,
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True
)
coll = COLLECT(
exe,
analysis.binaries,
analysis.zipfiles,
analysis.datas,
strip=False,
upx=True,
name='sam'
)
59 changes: 34 additions & 25 deletions installer/pyinstaller/samcli.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,38 @@
import sys ; sys.setrecursionlimit(sys.getrecursionlimit() * 5)
block_cipher = None
exe_name = 'sam'
analysis = Analysis(['../../samcli/__main__.py'],
binaries=[('/usr/local/lib/libcrypt.so.2', '.')],
datas=[],
hiddenimports=[],
hookspath=['./installer/pyinstaller'],
runtime_hooks=[],
excludes=[],
cipher=block_cipher)
analysis = Analysis(
['../../samcli/__main__.py'],
binaries=[
('/usr/local/lib/libbz2.so.1', '.'),
('/usr/local/lib/libcrypt.so.2', '.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was an earlier discussion with @hnnasit about removing this all together. Do we know if it is possible to remove these custom configs just like the MacOS ones?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, we can try removing both of these to test. We can add them back easily anyways

],
datas=[],
hiddenimports=[],
hookspath=['./installer/pyinstaller'],
runtime_hooks=[],
excludes=[],
cipher=block_cipher
)
pyz = PYZ(analysis.pure, analysis.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
analysis.scripts,
[],
exclude_binaries=True,
name=exe_name,
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
analysis.binaries,
analysis.zipfiles,
analysis.datas,
strip=False,
upx=True,
name='sam')
exe = EXE(
pyz,
analysis.scripts,
[],
exclude_binaries=True,
name=exe_name,
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True
)
coll = COLLECT(
exe,
analysis.binaries,
analysis.zipfiles,
analysis.datas,
strip=False,
upx=True,
name='sam'
)
Loading