Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docker-base-buster][docker-config-engine-buster] No longer install Python 2 #6162

Merged
merged 17 commits into from
Dec 26, 2020
Merged

[docker-base-buster][docker-config-engine-buster] No longer install Python 2 #6162

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
42c792b
[docker-base-buster] No longer install Python 2
jleveque Nov 10, 2020
ebf10a4
Remove Python 2 from docker-config-engine-buster
jleveque Nov 12, 2020
cfb9452
Install pip2 in PMon container
jleveque Dec 8, 2020
b3fe099
Minor tweaks to SNMP Dockerfile
jleveque Dec 8, 2020
67ffd8a
Install pip2 in docker-sonic-vs until we can remove entirely
jleveque Dec 8, 2020
ebf67a0
Install Python 3 version of sonic-cfggen in all containers, Py2 in PMon
jleveque Dec 9, 2020
b1f0b1d
No longer install Python 2 in docker-orchagent
jleveque Dec 9, 2020
6bad421
Install build-essential in PMon
jleveque Dec 9, 2020
1c88eaf
Additional cleanup; Install/upgrade pip, intstall setuptools and whee…
jleveque Dec 10, 2020
3207bdd
Add Python 2 back to docker-orchagent until restore_neighbors.py is P…
jleveque Dec 10, 2020
2728f3f
Add quotes around pip version
jleveque Dec 10, 2020
a156956
Add Python 2 back to docker-sonic-mgmt-framework until it is no longe…
jleveque Dec 11, 2020
7859787
Install python3-distutils in docker-base-buster
jleveque Dec 11, 2020
82b64b6
Install Python 2 setuptools and wheel packages in docker-sonic-mgmt-f…
jleveque Dec 11, 2020
5b84690
Fix new merge conflicts
jleveque Dec 15, 2020
b0f53d0
[docker-database-init.sh] Call python3 explicitly
jleveque Dec 18, 2020
6e8679b
[docker-database/supervisord.conf.j2] Change 'iteritems' to 'items' t…
jleveque Dec 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions dockers/docker-base-buster/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,8 @@ RUN apt-get update && \
less \
perl \
procps \
python \
python-pip \
python3 \
python3-distutils \
python3-pip \
rsyslog \
vim-tiny \
Expand All @@ -57,7 +56,6 @@ RUN apt-get update && \
lua-bitop \
lua-cjson \
# common dependencies
libpython2.7 \
libdaemon0 \
libdbus-1-3 \
libjansson4 \
Expand All @@ -84,15 +82,16 @@ RUN dpkg -i redis-tools_6.0.6-1~bpo10+1_amd64.deb || apt-get install -f -y
RUN rm redis-tools_6.0.6-1~bpo10+1_amd64.deb
{% endif %}

# Some Python packages require setuptools (or pkg_resources, which is supplied by setuptools)
# and some require wheel
RUN pip install setuptools==40.8.0
RUN pip3 install setuptools==49.6.00
RUN pip install wheel
RUN pip3 install wheel
# Upgrade pip via PyPI and uninstall the Debian version
RUN pip3 install --upgrade pip
RUN apt-get purge -y python3-pip

# setuptools and wheel are necessary for installing some Python wheel packages
RUN pip3 install --no-cache-dir setuptools==49.6.00
RUN pip3 install --no-cache-dir wheel==0.35.1

# For templating
RUN pip2 install j2cli
RUN pip3 install j2cli

# Install supervisor
RUN pip3 install supervisor==4.2.1
Expand Down
3 changes: 3 additions & 0 deletions dockers/docker-base/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ COPY ["etc/rsyslog.conf", "/etc/rsyslog.conf"]
COPY ["etc/rsyslog.d/*", "/etc/rsyslog.d/"]
COPY ["root/.vimrc", "/root/.vimrc"]

RUN pip install --upgrade 'pip<21'
RUN apt-get purge -y python-pip
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why to touch it.

Copy link
Contributor Author

@jleveque jleveque Dec 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In January, pip version 21 will be released, which will drop support for Python 2. Therefore, we need to ensure we install a version < 21 for Python 2 purposes.


# Some Python packages require setuptools (or pkg_resources, which is supplied by setuptools)
# and some require wheel
RUN pip install setuptools==40.8.0
Expand Down
18 changes: 5 additions & 13 deletions dockers/docker-config-engine-buster/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,15 @@ ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
apt-get install -y \
build-essential \
python-pip \
python3-pip \
python-dev \
python3-dev \
apt-utils \
build-essential \
python3-dev

{%- if CONFIGURED_ARCH == "armhf" or CONFIGURED_ARCH == "arm64" %}
RUN apt-get install -y \
libxslt-dev \
libz-dev \
libz-dev
{%- endif %}
python-setuptools \
python3-setuptools

RUN pip3 install --upgrade pip
RUN pip2 install --upgrade pip
RUN apt-get purge -y python-pip python3-pip

# For sonic-config-engine Python 3 package
# Explicitly install pyangbind here, as pyangbind causes enum34 to be installed.
Expand Down Expand Up @@ -51,7 +44,6 @@ COPY ["files/swss_vars.j2", "/usr/share/sonic/templates/"]

## Clean up
RUN apt-get purge -y \
python-dev \
python3-dev \
build-essential && \
apt-get clean -y && \
Expand Down
11 changes: 4 additions & 7 deletions dockers/docker-config-engine-stretch/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,13 @@ RUN apt-get update && \
apt-get install -y \
# Dependencies for sonic-cfggen
build-essential \
python-pip \
python-dev \
python-dev

{%- if CONFIGURED_ARCH == "armhf" or CONFIGURED_ARCH == "arm64" %}
RUN apt-get install -y \
libxslt-dev \
libz-dev \
libz-dev
{%- endif %}
python-setuptools

RUN pip install --upgrade pip
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why we need to modify stretch docker.

Copy link
Contributor Author

@jleveque jleveque Dec 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is simply to align the way we install/upgrade pip everywhere


{% if docker_config_engine_stretch_debs.strip() %}
# Copy locally-built Debian package dependencies
Expand All @@ -40,7 +38,6 @@ COPY ["files/swss_vars.j2", "/usr/share/sonic/templates/"]
## Clean up
RUN apt-get purge -y \
build-essential \
python-pip \
python-dev && \
apt-get clean -y && \
apt-get autoclean -y && \
Expand Down
6 changes: 1 addition & 5 deletions dockers/docker-config-engine/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update

# Dependencies for sonic-cfggen
RUN apt-get install -y build-essential python-pip python-dev

RUN pip install --upgrade pip

RUN apt-get purge -y python-pip
RUN apt-get install -y build-essential python-dev
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why we modify this file.

Copy link
Contributor Author

@jleveque jleveque Dec 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is simply to align the way we install/upgrade pip everywhere


{% if docker_config_engine_debs.strip() %}
COPY \
Expand Down
2 changes: 1 addition & 1 deletion dockers/docker-database/docker-database-init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ rm $db_cfg_file_tmp

# copy dump.rdb file to each instance for restoration
DUMPFILE=/var/lib/redis/dump.rdb
redis_inst_list=`/usr/bin/python -c "import swsssdk; print(' '.join(swsssdk.SonicDBConfig.get_instancelist().keys()))"`
redis_inst_list=`/usr/bin/python3 -c "import swsssdk; print(' '.join(swsssdk.SonicDBConfig.get_instancelist().keys()))"`
for inst in $redis_inst_list
do
mkdir -p /var/lib/$inst
Expand Down
2 changes: 1 addition & 1 deletion dockers/docker-database/supervisord.conf.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ stdout_logfile=syslog
stderr_logfile=syslog

{% if INSTANCES %}
{% for redis_inst, redis_items in INSTANCES.iteritems() %}
{% for redis_inst, redis_items in INSTANCES.items() %}
[program: {{ redis_inst }}]
{% if redis_items['hostname'] != '127.0.0.1' and redis_inst != 'redis_chassis' %}
{%- set LOOPBACK_IP = '127.0.0.1' -%}
Expand Down
10 changes: 0 additions & 10 deletions dockers/docker-orchagent/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ RUN apt-get update && \
ndppd \
# Needed for installing netifaces Python package
build-essential \
python-dev \
python3-dev

{% if ( CONFIGURED_ARCH == "armhf" or CONFIGURED_ARCH == "arm64" ) %}
Expand All @@ -36,14 +35,6 @@ RUN ln -s -f /usr/bin/gcc-8 /usr/bin/arm-linux-gnueabihf-gcc
RUN ln -s -f /usr/bin/gcc-8 /usr/bin/aarch64-linux-gnu-gcc
{% endif %}

RUN pip2 install \
scapy==2.4.2 \
setuptools \
pyroute2==0.5.3
RUN pip2 install \
netifaces==0.10.7 \
monotonic==1.5

# Dependencies of restore_neighbors.py
RUN pip3 install \
jleveque marked this conversation as resolved.
Show resolved Hide resolved
scapy==2.4.4 \
Expand All @@ -66,7 +57,6 @@ RUN apt-get remove -y gcc-8
# Clean up
RUN apt-get purge -y \
build-essential \
python-dev \
python3-dev && \
apt-get clean -y && \
apt-get autoclean -y && \
Expand Down
13 changes: 11 additions & 2 deletions dockers/docker-platform-monitor/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ ENV IMAGE_VERSION=$image_version
# Install required packages
RUN apt-get update && \
apt-get install -y \
python-dev \
build-essential \
python3-dev \
ipmitool \
librrd8 \
Expand All @@ -26,6 +26,11 @@ RUN apt-get update && \
i2c-tools \
python-jsonschema

# TODO: Remove these lines once we no longer need Python 2
RUN apt-get install -f -y python-dev python-pip
RUN pip2 install --upgrade 'pip<21'
RUN apt-get purge -y python-pip

# On Arista devices, the sonic_platform wheel is not installed in the container.
# Instead, the installation directory is mounted from the host OS. However, this method
# doesn't ensure all dependencies are installed in the container. So here we
Expand Down Expand Up @@ -64,9 +69,13 @@ RUN pip3 install thrift==0.13.0
{{ install_python_wheels(docker_platform_monitor_whls.split(' ')) }}
{% endif %}


# TODO: Remove this line once we no longer need Python 2
RUN apt-get purge -y python-dev

# Clean up
RUN apt-get purge -y \
python-dev \
build-essential \
python3-dev && \
apt-get clean -y && \
apt-get autoclean -y && \
Expand Down
8 changes: 2 additions & 6 deletions dockers/docker-snmp/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,8 @@ ENV LC_CTYPE=en_US.UTF-8
RUN sed -i '/^#.* en_US.* /s/^#//' /etc/locale.gen
RUN locale-gen

# Install up-to-date version of pip
RUN pip3 install --no-cache-dir setuptools wheel

# Install pyyaml dependency for use by some plugins
# Install smbus dependency for use by some plugins
RUN python3 -m pip install --no-cache-dir \
# Install dependencies used by some plugins
RUN pip3 install --no-cache-dir \
hiredis \
pyyaml \
smbus
Expand Down
15 changes: 12 additions & 3 deletions dockers/docker-sonic-mgmt-framework/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,14 @@ RUN [ -f /etc/rsyslog.conf ] && sed -ri "s/%syslogtag%/$docker_container_name#%s
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
apt-get install -y g++ python-dev python3-dev libxml2

apt-get install -y g++ python3-dev libxml2

# TODO: Remove these lines once we no longer need Python 2
RUN apt-get install -f -y python-dev python-pip
RUN pip2 install --upgrade 'pip<21'
RUN apt-get purge -y python-pip
RUN pip2 install setuptools==40.8.0
RUN pip2 install wheel==0.35.1
RUN pip2 install connexion==1.1.15 \
setuptools==21.0.0 \
grpcio-tools==1.20.0 \
Expand Down Expand Up @@ -40,7 +46,10 @@ COPY ["start.sh", "rest-server.sh", "/usr/bin/"]
COPY ["mgmt_vars.j2", "/usr/share/sonic/templates/"]
COPY ["supervisord.conf", "/etc/supervisor/conf.d/"]

RUN apt-get remove -y g++ python-dev python3-dev
# TODO: Remove this line once we no longer need Python 2
RUN apt-get purge -y python-dev

RUN apt-get remove -y g++ python3-dev
RUN apt-get clean -y; apt-get autoclean -y; apt-get autoremove -y
RUN rm -rf /debs

Expand Down
8 changes: 5 additions & 3 deletions platform/vs/docker-sonic-vs/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,13 @@ RUN apt-get install -y net-tools \
python-scapy \
conntrack \
iptables \
python3-pip \
jq \
libzmq5 \
# For installing Python m2crypto package
# (these can be uninstalled after installation)
build-essential \
python-dev \
python-pip \
python3-dev \
libssl-dev \
swig \
Expand All @@ -73,8 +73,10 @@ RUN dpkg -i redis-tools_6.0.6-1~bpo10+1_amd64.deb redis-server_6.0.6-1~bpo10+1_a
RUN rm redis-tools_6.0.6-1~bpo10+1_amd64.deb redis-server_6.0.6-1~bpo10+1_amd64.deb
{% endif %}

RUN pip2 install setuptools
RUN pip3 install setuptools
RUN pip2 install --upgrade 'pip<21'
RUN apt-get purge -y python-pip
RUN pip2 install setuptools==40.8.0
RUN pip2 install wheel==0.33.6
RUN pip2 install py2_ipaddress
RUN pip2 install six
RUN pip2 install pyroute2==0.5.3 netifaces==0.10.7
Expand Down
3 changes: 0 additions & 3 deletions rules/docker-config-engine-buster.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,8 @@ DOCKER_CONFIG_ENGINE_BUSTER = docker-config-engine-buster.gz
$(DOCKER_CONFIG_ENGINE_BUSTER)_PATH = $(DOCKERS_PATH)/docker-config-engine-buster

$(DOCKER_CONFIG_ENGINE_BUSTER)_DEPENDS += $(LIBSWSSCOMMON) $(PYTHON3_SWSSCOMMON)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SWSSSDK_PY2)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SWSSSDK_PY3)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY2)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY2)
$(DOCKER_CONFIG_ENGINE_BUSTER)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY3)
$(DOCKER_CONFIG_ENGINE_BUSTER)_LOAD_DOCKERS += $(DOCKER_BASE_BUSTER)
$(DOCKER_CONFIG_ENGINE_BUSTER)_FILES += $(SWSS_VARS_TEMPLATE)
Expand Down
1 change: 1 addition & 0 deletions rules/docker-platform-monitor.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ $(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SWSSSDK_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SWSSSDK_PY3)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PY_COMMON_PY3)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_CONFIG_ENGINE_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PLATFORM_API_PY2)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_PLATFORM_API_PY3)
$(DOCKER_PLATFORM_MONITOR)_PYTHON_WHEELS += $(SONIC_LEDD)
Expand Down
2 changes: 1 addition & 1 deletion sonic-slave-buster/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ RUN export VERSION=1.14.2 \
&& rm go$VERSION.linux-*.tar.gz

RUN pip3 install --upgrade pip
RUN pip2 install --upgrade pip
RUN pip2 install --upgrade 'pip<21'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why need this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In January, pip version 21 will be released, which will drop support for Python 2. Therefore, we need to ensure we install a version < 21 for Python 2 purposes.

RUN apt-get purge -y python-pip python3-pip

# For building Python packages
Expand Down
2 changes: 1 addition & 1 deletion sonic-slave-stretch/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ RUN export VERSION=1.14.2 \
&& rm go$VERSION.linux-*.tar.gz

RUN pip3 install --upgrade pip
RUN pip2 install --upgrade pip
RUN pip2 install --upgrade 'pip<21'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why need this

Copy link
Contributor Author

@jleveque jleveque Dec 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In January, pip version 21 will be released, which will drop support for Python 2. Therefore, we need to ensure we install a version < 21 for Python 2 purposes.

RUN apt-get purge -y python-pip python3-pip

# For p4 build
Expand Down