Data mover micro service smoke testing

[Lyndon-Li]

Enhance some of the data mover micro service code and fix the problems found during smoke testing

[codecov]

Codecov Report

Attention: Patch coverage is 63.88889% with 65 lines in your changes missing coverage. Please review.

Project coverage is 59.08%. Comparing base (8fde4a0) to head (0ed1a7f).
Report is 17 commits behind head on main.

Files	Patch %	Lines
pkg/datapath/micro_service_watcher.go	36.11%	21 Missing and 2 partials ⚠️
pkg/cmd/cli/nodeagent/server.go	0.00%	22 Missing ⚠️
pkg/datapath/file_system.go	72.72%	6 Missing ⚠️
pkg/controller/data_download_controller.go	89.13%	5 Missing ⚠️
pkg/controller/pod_volume_backup_controller.go	0.00%	3 Missing ⚠️
pkg/controller/pod_volume_restore_controller.go	0.00%	3 Missing ⚠️
pkg/exposer/csi_snapshot.go	0.00%	2 Missing ⚠️
pkg/controller/data_upload_controller.go	97.36%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8115      +/-   ##
==========================================
+ Coverage   59.02%   59.08%   +0.06%     
==========================================
  Files         364      364              
  Lines       30272    30300      +28     
==========================================
+ Hits        17867    17902      +35     
+ Misses      10959    10955       -4     
+ Partials     1446     1443       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shubham-pampattiwar]

Wouldn't calling closeDataPath be a better choice instead of directly calling RemoveAsyncBR ? Why are we changing this from earlier behavior ? (same question for datadownload/pvb/pvr)

[Lyndon-Li]

It is under the context of callbacks which are called by the async backup/restore thread.
So calling closeDataPath which calls asyncBR.Close is not literally correct --- the async backup/restore thread is a part of the asyncBR instance so asyncBR.Close literally needs to first wait the thread and then close all the leftovers. However, this will cause a deadlock.
This is why we needed a go routine to call closeDataPath in the callbacks before this change. However, that is not a ideal solution.

Therefore, the logics are refactored:

• We create two functions for asyncBR, close and Close. The latter one calls the former one
• The async backup/restore thread calls close by the end itself
• So the callbacks never need to care about the closure of the asyncBR, they only need to take care of their own data
• Close is used by the caller to stop asyncBR outside of the async thread