Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dockerfile lock gets confused when installing pip package with an OS package manager #702

Closed
rnjudge opened this issue May 26, 2020 · 0 comments · Fixed by #707
Closed

Dockerfile lock gets confused when installing pip package with an OS package manager #702

rnjudge opened this issue May 26, 2020 · 0 comments · Fixed by #707
Assignees
@rnjudge
Labels
bug Something went wrong
Milestone
Release 2.1.0

Comments

@rnjudge
Copy link
Contributor

rnjudge commented May 26, 2020

Describe the bug
In Tern's Dockerfile we apt-install the pip package like so:

RUN tdnf remove -y toybox && tdnf install -y tar findutils attr util-linux python3
python3-pip python3-setuptools git && pip3 install --upgrade pip && pip3 install tern

To Reproduce
Steps to reproduce the behavior:

  1. Run tern lock Dockerfile at the tip of master.
  2. See resulting Dockerfile.lock file.

Error in terminal
The resulting Dockerfile.lock incorrectly pins the python3-pip package like so: python3-pip==20.1.1==20.1.1-3.7.5-3.ph3

RUN tdnf remove -y toybox && tdnf install -y tar-1.30-4.ph3 findutils-4.6.0-5.ph3
 attr-2.4.48-1.ph3 util-linux-2.32.1-2.ph3 python3-3.7.5-3.ph3 python3-
pip==20.1.1==20.1.1-3.7.5-3.ph3 python3-setuptools-3.7.5-3.ph3 git-2.23.3-1.ph3 && pip3 
install --upgrade pip && pip3 install tern==2.0.

Expected behavior
The python3-pip package should be pinned properly.
@rnjudge rnjudge added the bug Something went wrong label May 26, 2020
@rnjudge rnjudge self-assigned this May 26, 2020
rnjudge added a commit to rnjudge/tern that referenced this issue May 28, 2020
@rnjudge
Fix package pinning when package is also pkg mngr
1643996 
Since the addition of language package managers, the Dockerfile lock
functionality was getting confused when trying to pin packages that also
double as language package managers.

This commit does the following to resolve the issue:
 1) In analyze.py, instead of comparing each of the packages in the
    layer to see if they are installed in the RUN line, first gather
    the list of packages being installed in the dockerfile RUN line
    and then compare those against the packages installed in the layer.
    This prevents unnecessary comparisons and duplicate pinnings.

 2) Add a 'should_pin' method to dockerfile.py that checks whether the
    package in the RUN line should be pinned or not. Use this function
    in the existing 'expand_package' function in dockerfile.py to
    determine if we should expand/pin the package or not.

 3) Modify the way 'expand_package' in dockerfile.py actually pins the
    package and version in the dockerfile object. Instead of using the
    replace method, which was expanding non-exact matches of a package
    name, we go word by word in the RUN line to determine if a package
    should be pinned.

  4) Make the command_lib dictionary global so we can access snippet
     information in dockerfile.py

  5) Modify the 'package_in_dockerfile' function to return the list of
     packages being installed in a given RUN line instead of returning
     true or false. Also change the name of the function to more
     accurately reflect its purpose. The function is now called
     'get_install_packages'.

Resolves tern-tools#702

Signed-off-by: Rose Judge <rjudge@vmware.com>
@rnjudge rnjudge mentioned this issue May 28, 2020
Merged
rnjudge added a commit to rnjudge/tern that referenced this issue May 28, 2020
@rnjudge
Fix package pinning when package is also pkg mngr
6b2a79b 
Since the addition of language package managers, the Dockerfile lock
functionality was getting confused when trying to pin packages that also
double as language package managers.

This commit does the following to resolve the issue:
 1) In analyze.py, instead of comparing each of the packages in the
    layer to see if they are installed in the RUN line, first gather
    the list of packages being installed in the dockerfile RUN line
    and then compare those against the packages installed in the layer.
    This prevents unnecessary comparisons and duplicate pinnings.

 2) Add a 'should_pin' method to dockerfile.py that checks whether the
    package in the RUN line should be pinned or not. Use this function
    in the existing 'expand_package' function in dockerfile.py to
    determine if we should expand/pin the package or not.

 3) Modify the way 'expand_package' in dockerfile.py actually pins the
    package and version in the dockerfile object. Instead of using the
    replace method, which was expanding non-exact matches of a package
    name, we go word by word in the RUN line to determine if a package
    should be pinned.

  4) Modify the 'package_in_dockerfile' function to return the list of
     packages being installed in a given RUN line instead of returning
     true or false. Also change the name of the function to more
     accurately reflect its purpose. The function is now called
     'get_install_packages'.

Resolves tern-tools#702

Signed-off-by: Rose Judge <rjudge@vmware.com>
nishakm pushed a commit that referenced this issue May 28, 2020
@rnjudge
Fix package pinning when package is also pkg mngr
202c4c8 
Since the addition of language package managers, the Dockerfile lock
functionality was getting confused when trying to pin packages that also
double as language package managers.

This commit does the following to resolve the issue:
 1) In analyze.py, instead of comparing each of the packages in the
    layer to see if they are installed in the RUN line, first gather
    the list of packages being installed in the dockerfile RUN line
    and then compare those against the packages installed in the layer.
    This prevents unnecessary comparisons and duplicate pinnings.

 2) Add a 'should_pin' method to dockerfile.py that checks whether the
    package in the RUN line should be pinned or not. Use this function
    in the existing 'expand_package' function in dockerfile.py to
    determine if we should expand/pin the package or not.

 3) Modify the way 'expand_package' in dockerfile.py actually pins the
    package and version in the dockerfile object. Instead of using the
    replace method, which was expanding non-exact matches of a package
    name, we go word by word in the RUN line to determine if a package
    should be pinned.

  4) Modify the 'package_in_dockerfile' function to return the list of
     packages being installed in a given RUN line instead of returning
     true or false. Also change the name of the function to more
     accurately reflect its purpose. The function is now called
     'get_install_packages'.

Resolves #702

Signed-off-by: Rose Judge <rjudge@vmware.com>
@rnjudge rnjudge added this to the Release 2.1.0 milestone May 28, 2020
rnjudge added a commit to rnjudge/tern that referenced this issue Jun 5, 2020
@rnjudge
Fix package pinning when package is also pkg mngr
78e6890 
Since the addition of language package managers, the Dockerfile lock
functionality was getting confused when trying to pin packages that also
double as language package managers.

This commit does the following to resolve the issue:
 1) In analyze.py, instead of comparing each of the packages in the
    layer to see if they are installed in the RUN line, first gather
    the list of packages being installed in the dockerfile RUN line
    and then compare those against the packages installed in the layer.
    This prevents unnecessary comparisons and duplicate pinnings.

 2) Add a 'should_pin' method to dockerfile.py that checks whether the
    package in the RUN line should be pinned or not. Use this function
    in the existing 'expand_package' function in dockerfile.py to
    determine if we should expand/pin the package or not.

 3) Modify the way 'expand_package' in dockerfile.py actually pins the
    package and version in the dockerfile object. Instead of using the
    replace method, which was expanding non-exact matches of a package
    name, we go word by word in the RUN line to determine if a package
    should be pinned.

  4) Modify the 'package_in_dockerfile' function to return the list of
     packages being installed in a given RUN line instead of returning
     true or false. Also change the name of the function to more
     accurately reflect its purpose. The function is now called
     'get_install_packages'.

Resolves tern-tools#702

Signed-off-by: Rose Judge <rjudge@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rnjudge rnjudge

Labels
bug Something went wrong
Projects
None yet
Milestone
Release 2.1.0
Development

Successfully merging a pull request may close this issue.

Fix package pinning when package is also pkg mngr rnjudge/tern
1 participant
@rnjudge