[cfggen] Support reading from and writing to configdb #808

taoyl-ms · 2017-07-13T21:30:01Z

No description provided.

lguohan · 2017-07-14T02:16:04Z

retest this please #Closed

lguohan · 2017-07-16T00:45:51Z

can you do a rebase so that it is easy to check the changes from codeflow. #Resolved

…xist

taoyl-ms · 2017-07-16T19:22:45Z

done

In reply to: 315573216 [](ancestors = 315573216)

lguohan · 2017-07-19T17:39:25Z

dockers/docker-database/start.sh

+            break
+        fi
+        sleep 1
+    done


while true; do if [ "$(redis-cli ping)" == "PONG" ]; then break; fi; sleep 1; done

is more reliable then just check existence of redis.sock #Resolved

lguohan · 2017-07-19T17:59:04Z

files/build_templates/sonic_debian_extension.j2

@@ -132,8 +132,7 @@ sudo bash -c "echo dhcp_as_static=true >> $FILESYSTEM_ROOT/etc/sonic/updategraph
 sudo bash -c "echo enabled=false > $FILESYSTEM_ROOT/etc/sonic/updategraph.conf"
 {% endif %}
 {% if shutdown_bgp_on_start == "y" %}
-sudo bash -c "echo bgp_admin_state: > $FILESYSTEM_ROOT/etc/sonic/bgp_admin.yml"
-sudo bash -c "echo '  all: off' >> $FILESYSTEM_ROOT/etc/sonic/bgp_admin.yml"
+sudo bash -c "echo '{ \"bgp_admin_state\": { \"all\": false }  }' >> $FILESYSTEM_ROOT/etc/sonic/config_db.json"
 {% endif %}


when the option is n, we should probably put bgp_admin_state to true. the reason is that when bgp docker start, it needs to read the bgp_admin_state, there is chance that the configuration is loaded to the config_db yet. In this case, how do you know whether config is already loaded but the state is true versus the config is not yet loaded? #ByDesign

I don't think it's a good idea to tell whether configuration is loaded or not by looking at the content of config. Instead, we should find other ways to ensure initial config loading is finished before other container consume DB data, either by systemd service sequence or by setting flag in DB.

In reply to: 128321125 [](ancestors = 128321125)

lguohan · 2017-07-19T18:03:49Z

src/sonic-config-engine/sonic-cfggen

    parser.add_argument("-a", "--additional-data", help="addition data, in json string")
+    parser.add_argument("-d", "--from-db", help="read config from configdb", action='store_true')


do we need this option, we should always read from the configdb. can we make it as default? #WontFix

I'll suggest keep the -d option during migration from minigraph to db. We can come back to revisit this after we move all data to DB.

In reply to: 128322480 [](ancestors = 128322480)

lguohan · 2017-07-19T18:24:21Z

src/sonic-config-engine/sonic-cfggen

    group = parser.add_mutually_exclusive_group()
    group.add_argument("-t", "--template", help="render the data with the template file")
    group.add_argument("-v", "--var", help="print the value of a variable, support jinja2 expression")
    group.add_argument("--var-json", help="print the value of a variable, in json format")
-    group.add_argument("--print-data", help="print all data", action='store_true')
+    group.add_argument("-D", "--write-to-db", help="write config into configdb", action='store_true')


-write-to-db [](start = 31, length = 12)

--import : import json into configdb #WontFix

Let's keep the current option setting for dev purposes. I'll later add a "config load" and "config save" command to CLI for users.

In reply to: 128327833 [](ancestors = 128327833)

lguohan · 2017-07-19T18:25:26Z

src/sonic-config-engine/sonic-cfggen

+    Dump config DB content into json file: 
+        sonic-cfggen -d --print-data > db_dump.json
+    Load content of json file into config DB:
+        sonic-cfggen -j db_dump.json --write-to-db


onic-cfggen -j db_dump.json --write-to-db [](start = 9, length = 41)

sonic-cfggen --import db_dump.json #WontFix

Let's keep the current option setting for dev purposes. I'll later add a "config load" and "config save" command to CLI for users.

In reply to: 128328144 [](ancestors = 128328144)

lguohan · 2017-07-19T18:26:19Z

src/sonic-config-engine/sonic-cfggen

+    Render template with minigraph:        
+        sonic-cfggen -m -t /usr/share/template/bgpd.conf.j2
+    Dump config DB content into json file: 
+        sonic-cfggen -d --print-data > db_dump.json


-d [](start = 21, length = 3)

if we assume by default it will load configdb, then we do not need this option. #WontFix

Before we fully remove minigraph dependency, let's keep -m and -d options.

In reply to: 128328361 [](ancestors = 128328361)

lguohan · 2017-07-19T18:27:10Z

🕐

lguohan · 2017-08-01T07:26:33Z

dockers/docker-fpm-quagga/start.sh

@@ -21,6 +18,8 @@ echo "# Config files managed by sonic-config-engine" > /var/sonic/config_status

 rm -f /var/run/rsyslogd.pid

+supervisorctl start bgpcfgd


probably need this for docker-fpm-frr too. #WontFix

Will resolve in later PRs.

In reply to: 130536078 [](ancestors = 130536078)

lguohan · 2017-08-01T07:38:10Z

dockers/docker-database/supervisord.conf

-[program:redis-server]
-command=/usr/bin/redis-server /etc/redis/redis.conf
+[program:start.sh]
+command=/usr/bin/start.sh


I do not why start redis-server inside start.sh, why can you make start.sh independent of redis-server start. maybe rename it as configdb-load.sh? #Resolved

It's just because we are following the similar design - start.sh controls all - in other dockers. As redis-server in docker-database does not have any dependency, I can decouple it with start.sh.

In reply to: 130537997 [](ancestors = 130537997)

lguohan

have a question on docker-database start.sh. frr support can be another pr.

taoyl-ms · 2017-08-02T00:10:08Z

retest this please #Closed

Update sonic-sairedis submodule and also update sonic-swss submodule as there are interdependent changes. * src/sonic-sairedis 13474d1...bc58b0f (12): > Add gbsyncdmgrd; deprecate gbsyncd_startup.py (#809) > Remove gbsyncd_start.sh (#808) > [gbsyncd] Fix shebang in gbsyncd_startup.py; Make script executable (#807) > [saiasiccmp] Add saiasiccmp tool to compare 2 asic views (#791) > [configure] Add -Wno-psabi to remove "passing argument changed in GCC 7.1" (#799) > Update FlexCounter.cpp, use m_pollInterval in MUTEX lock (#797) > [vs] Add special warm boot logic to populate default attributes (#796) > [ci]: add vstest (#795) > [tests] Add macsec unittest (#782) > [debian/control] libsairedis-dev depends on libzmq5-dev (#794) > [ci]: use build template (#793) > Rename duplicate file name (#773) * src/sonic-swss 0b0d24c...5adb73e (47): > Initialize system port type variable (#1681) > [Dynamic Buffer Calc] Enhance the field checking in table handling (#1680) > Handle the clear request for 'Q_SHARED_ALL' (#1653) > [MuxOrch] FDB ageout safety check (#1674) > Deactivate mirror session only when session status is true in updateLagMember (#1666) > Revert "[buffermgr] Support maximum port headroom checking (#1607)" (#1675) > reduce severity of log to info in case of flush on non-existing member (#1669) > Revert "[Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (#1652)" (#1676) > [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (#1652) > [acl] Move ACL table constants to acltable.h (#1671) > [nbrmgrd] added function to parse IP address from APP_DB (#1672) > [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (#1670) > [vog/systemlag] Voq lagid allocator (#1603) > Add table descriptions for dynamic buffer calculation to the documents (#1664) > [vstest/subintf] Add vs test case to validate processing sequence of APPL DB keys (#1663) > Remove vxlanmgrd dependency on orchagent (#1647) > Keep attribute order in bulk mode (#1659) > [mux] VS test for neigh, route and fdb (#1656) > [linksync] Netdev oper status determination using IFF_RUNNING (#1568) > [portorch] parse on/off value from autoneg (#1658) > [intfsorch] Create subport with the entry contains necessary attributes (#1650) > [ci]: Purge swss before install (#1654) > Update StateDB with error if state change failed, Update APP_DB in all state chg req (#1662) > Added changes to handle dependency check in FdbSyncd and FpmSyncd for warm-boot (#1556) > [synchronous mode] Add failure notification for SAI failures in synchronous mode (#1596) > [acl] Enable VLAN ID qualifier for ACL rules (#1648) > Updated PFCWD to use single ACL table for PFCWD and MUX (#1620) > [orchagent] Increase SAI REDIS response timeout to support FW upgrade during init (Mellanox only). (#1637) > [vstest/nhg]: use dvs_route fixture to make test_nhg more robust > [vstest]: add dvs_route fixture > [vstest/subintf] Update vs tests to validate physical port host interface vlan tag attribute (#1634) > Remove useless header in macsecorch (#1628) > Add SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS counter, create new FlexCounter group (#1600) > fixed unsupported resource issue (#1641) > [test_virtual_chassis]: use wait_for to make test more robust (#1640) > spell check fixes (#1630) > [bufferorch] Handle NOT IMPLEMENTED status returned during set attr operation (#1639) > [ci]: run vstest > [test_virtual_chassis]: use wait_for function to improve test robustness > [Mux] Neighbor handling based on FDB entry (#1631) > [ci]: use build template (#1633) > Log level change from ERR to INFO for fetch systemports issue (#1632) > Migrate serdes programming to port serdes object (#1611) > [tests] Remove legacy saiattributelist.h dependency (#1608) > [buffermgr] Support maximum port headroom checking (#1607) > Support shared headroom pool on top of dynamic buffer calculation (#1581) > Fix the compiling errors in gcc9 (#1621)

…ontainer auto-restart feature. (sonic-net#808) * [Command-reference.md] Correct a typo. Signed-off-by: Yong Zhao <yozhao@microsoft.com> * [Command-Reference.md] Fix the link jump. Signed-off-by: Yong Zhao <yozhao@microsoft.com> * [Command-Reference.md] Fix the link jump. Signed-off-by: Yong Zhao <yozhao@microsoft.com> * [Command-Reference.md] Fix the link jump. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Update sonic-sairedis submodule and also update sonic-swss submodule as there are interdependent changes. * src/sonic-sairedis 13474d1...bc58b0f (12): > Add gbsyncdmgrd; deprecate gbsyncd_startup.py (sonic-net#809) > Remove gbsyncd_start.sh (sonic-net#808) > [gbsyncd] Fix shebang in gbsyncd_startup.py; Make script executable (sonic-net#807) > [saiasiccmp] Add saiasiccmp tool to compare 2 asic views (sonic-net#791) > [configure] Add -Wno-psabi to remove "passing argument changed in GCC 7.1" (sonic-net#799) > Update FlexCounter.cpp, use m_pollInterval in MUTEX lock (sonic-net#797) > [vs] Add special warm boot logic to populate default attributes (sonic-net#796) > [ci]: add vstest (sonic-net#795) > [tests] Add macsec unittest (sonic-net#782) > [debian/control] libsairedis-dev depends on libzmq5-dev (sonic-net#794) > [ci]: use build template (sonic-net#793) > Rename duplicate file name (sonic-net#773) * src/sonic-swss 0b0d24c...5adb73e (47): > Initialize system port type variable (sonic-net#1681) > [Dynamic Buffer Calc] Enhance the field checking in table handling (sonic-net#1680) > Handle the clear request for 'Q_SHARED_ALL' (sonic-net#1653) > [MuxOrch] FDB ageout safety check (sonic-net#1674) > Deactivate mirror session only when session status is true in updateLagMember (sonic-net#1666) > Revert "[buffermgr] Support maximum port headroom checking (sonic-net#1607)" (sonic-net#1675) > reduce severity of log to info in case of flush on non-existing member (sonic-net#1669) > Revert "[Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)" (sonic-net#1676) > [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652) > [acl] Move ACL table constants to acltable.h (sonic-net#1671) > [nbrmgrd] added function to parse IP address from APP_DB (sonic-net#1672) > [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (sonic-net#1670) > [vog/systemlag] Voq lagid allocator (sonic-net#1603) > Add table descriptions for dynamic buffer calculation to the documents (sonic-net#1664) > [vstest/subintf] Add vs test case to validate processing sequence of APPL DB keys (sonic-net#1663) > Remove vxlanmgrd dependency on orchagent (sonic-net#1647) > Keep attribute order in bulk mode (sonic-net#1659) > [mux] VS test for neigh, route and fdb (sonic-net#1656) > [linksync] Netdev oper status determination using IFF_RUNNING (sonic-net#1568) > [portorch] parse on/off value from autoneg (sonic-net#1658) > [intfsorch] Create subport with the entry contains necessary attributes (sonic-net#1650) > [ci]: Purge swss before install (sonic-net#1654) > Update StateDB with error if state change failed, Update APP_DB in all state chg req (sonic-net#1662) > Added changes to handle dependency check in FdbSyncd and FpmSyncd for warm-boot (sonic-net#1556) > [synchronous mode] Add failure notification for SAI failures in synchronous mode (sonic-net#1596) > [acl] Enable VLAN ID qualifier for ACL rules (sonic-net#1648) > Updated PFCWD to use single ACL table for PFCWD and MUX (sonic-net#1620) > [orchagent] Increase SAI REDIS response timeout to support FW upgrade during init (Mellanox only). (sonic-net#1637) > [vstest/nhg]: use dvs_route fixture to make test_nhg more robust > [vstest]: add dvs_route fixture > [vstest/subintf] Update vs tests to validate physical port host interface vlan tag attribute (sonic-net#1634) > Remove useless header in macsecorch (sonic-net#1628) > Add SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS counter, create new FlexCounter group (sonic-net#1600) > fixed unsupported resource issue (sonic-net#1641) > [test_virtual_chassis]: use wait_for to make test more robust (sonic-net#1640) > spell check fixes (sonic-net#1630) > [bufferorch] Handle NOT IMPLEMENTED status returned during set attr operation (sonic-net#1639) > [ci]: run vstest > [test_virtual_chassis]: use wait_for function to improve test robustness > [Mux] Neighbor handling based on FDB entry (sonic-net#1631) > [ci]: use build template (sonic-net#1633) > Log level change from ERR to INFO for fetch systemports issue (sonic-net#1632) > Migrate serdes programming to port serdes object (sonic-net#1611) > [tests] Remove legacy saiattributelist.h dependency (sonic-net#1608) > [buffermgr] Support maximum port headroom checking (sonic-net#1607) > Support shared headroom pool on top of dynamic buffer calculation (sonic-net#1581) > Fix the compiling errors in gcc9 (sonic-net#1621)

`gbsyncd_start.sh` was a simple wrapper which only called `exec "/usr/bin/gbsyncd_startup.py"`. This causes unnecessary extra complexity and maintenance. It makes things simpler to run `/usr/bin/gbsyncd_startup.py` directly, which is being done by sonic-net#7084. That PR has merged, so we can now merge this.

…omatically (#16696) #### Why I did it src/sonic-swss-common ``` * b0f148e - (HEAD -> master, origin/master, origin/HEAD) [chassis][voq] Add fabric monitoring tables definitions. (#808) (10 hours ago) [jfeng-arista] ``` #### How I did it #### How to verify it #### Description for the changelog

taoyl-ms requested a review from lguohan July 13, 2017 21:30

msftclas added the cla-not-required label Jul 13, 2017

jleveque added Enhancement ➕ Submodule Update ⏫ labels Jul 14, 2017

taoyl-ms added 5 commits July 16, 2017 12:19

Support -j to read json file in cfggen

4f53ec1

[cfggen] Support reading from and writing to configdb

41bfa76

Do not create bgp_admin_state var when corresponding table does not e…

419e036

…xist

Move bgp_admin_state to configdb

79f1e17

Fix comments

76b888e

taoyl-ms force-pushed the taoyl/cfggen branch from 3573ecd to 76b888e Compare July 16, 2017 19:21

lguohan reviewed Jul 19, 2017

View reviewed changes

taoyl-ms added 5 commits July 20, 2017 00:48

[bgpd] Support dynamic admin status change

49f7143

Resolve CR comments

6b4063e

Fix logic for configdb initialization

16685db

Update sonic-utilities pointer

6e92563

Update sonic-py-swsssdk pointer

e092256

lguohan reviewed Aug 1, 2017

View reviewed changes

Decouple start.sh and redis-server in docker-database

93c3495

lguohan approved these changes Aug 2, 2017

View reviewed changes

taoyl-ms merged commit b6efe43 into sonic-net:master Aug 2, 2017

abdosi mentioned this pull request Feb 21, 2020

Made Change to have Configurable option to enable/disable docker image #4171

Merged

jleveque mentioned this pull request Mar 22, 2021

[sonic-sairedis][sonic-swss] Update submodules #7089

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cfggen] Support reading from and writing to configdb #808

[cfggen] Support reading from and writing to configdb #808

taoyl-ms commented Jul 13, 2017

lguohan commented Jul 14, 2017 •

edited by taoyl-ms

Loading

lguohan commented Jul 16, 2017 •

edited by taoyl-ms

Loading

taoyl-ms commented Jul 16, 2017

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Jul 19, 2017

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Jul 20, 2017

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Jul 20, 2017

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Jul 20, 2017

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Jul 20, 2017

lguohan commented Jul 19, 2017

lguohan Aug 1, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Aug 1, 2017

lguohan Aug 1, 2017 •

edited by taoyl-ms

Loading

taoyl-ms Aug 1, 2017

lguohan left a comment

taoyl-ms commented Aug 2, 2017 •

edited

Loading

		parser.add_argument("-a", "--additional-data", help="addition data, in json string")
		parser.add_argument("-d", "--from-db", help="read config from configdb", action='store_true')

		@@ -21,6 +18,8 @@ echo "# Config files managed by sonic-config-engine" > /var/sonic/config_status

		rm -f /var/run/rsyslogd.pid

		supervisorctl start bgpcfgd

[cfggen] Support reading from and writing to configdb #808

[cfggen] Support reading from and writing to configdb #808

Conversation

taoyl-ms commented Jul 13, 2017

lguohan commented Jul 14, 2017 • edited by taoyl-ms Loading

lguohan commented Jul 16, 2017 • edited by taoyl-ms Loading

taoyl-ms commented Jul 16, 2017

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Jul 19, 2017

Choose a reason for hiding this comment

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Jul 20, 2017

Choose a reason for hiding this comment

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Jul 20, 2017

Choose a reason for hiding this comment

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Jul 20, 2017

Choose a reason for hiding this comment

lguohan Jul 19, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Jul 20, 2017

Choose a reason for hiding this comment

lguohan commented Jul 19, 2017

lguohan Aug 1, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Aug 1, 2017

Choose a reason for hiding this comment

lguohan Aug 1, 2017 • edited by taoyl-ms Loading

Choose a reason for hiding this comment

taoyl-ms Aug 1, 2017

Choose a reason for hiding this comment

lguohan left a comment

Choose a reason for hiding this comment

taoyl-ms commented Aug 2, 2017 • edited Loading

lguohan commented Jul 14, 2017 •

edited by taoyl-ms

Loading

lguohan commented Jul 16, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Jul 19, 2017 •

edited by taoyl-ms

Loading

lguohan Aug 1, 2017 •

edited by taoyl-ms

Loading

lguohan Aug 1, 2017 •

edited by taoyl-ms

Loading

taoyl-ms commented Aug 2, 2017 •

edited

Loading