Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support hostNetwork #774

Merged
merged 6 commits into from
Aug 21, 2019
Merged

Support hostNetwork #774

merged 6 commits into from
Aug 21, 2019

Conversation

cofyc
Copy link
Contributor

@cofyc cofyc commented Aug 15, 2019 •
edited
Loading

What problem does this PR solve?

This adds a new filed in PDSpec/TiKVSpec/TiDBSpec to make it possible to run TiDB components (pd, tikv, tidb) in the host network.

fix #759

What is changed and how does it work?

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Code changes

  • Has Helm charts change
  • Has Go code change

Does this PR introduce a user-facing change?:

Add `hostNetwork` field in pd/tikv/tidb spec to make it possible to run tidb components in host network.

cofyc
cofyc commented Aug 15, 2019
View reviewed changes
pkg/apis/pingcap.com/v1alpha1/types.go Outdated
Tolerations []corev1.Toleration `json:"tolerations,omitempty"`
Annotations map[string]string `json:"annotations,omitempty"`
HostNetwork bool `json:"hostNetwork,omitempty"`
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PodSpec is created to simplify code only. Only one field hostNetwork is introduced.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been reverted because it cannot pass go vet check.

pkg/apis/pingcap.com/v1alpha1/tidbcluster_test.go:180:5: cannot use promoted field PodSpec.Replicas in struct literal of type PDSpec
pkg/apis/pingcap.com/v1alpha1/tidbcluster_test.go:183:5: cannot use promoted field PodSpec.Replicas in struct literal of type TiKVSpec
pkg/apis/pingcap.com/v1alpha1/tidbcluster_test.go:186:5: cannot use promoted field PodSpec.Replicas in struct literal of type TiDBSpec
...

cofyc
cofyc commented Aug 15, 2019
View reviewed changes
pkg/manager/member/tikv_member_manager.go
@@ -320,6 +320,11 @@ func (tkmm *tikvMemberManager) getNewSetForTidbCluster(tc *v1alpha1.TidbCluster)
storageClassName = controller.DefaultStorageClassName
}

dnsPolicy := corev1.DNSClusterFirst // same as k8s defaults
if tc.Spec.PD.HostNetwork {
dnsPolicy = corev1.DNSClusterFirstWithHostNet
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DNSClusterFirstWithHostNet is required in the host network, see https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy. Otherwise, the DNS resolver is the same as the host. Our components cannot resolve Kubernetes DNS names.

cofyc
cofyc commented Aug 15, 2019
View reviewed changes
charts/tidb-cluster/templates/scripts/_start_pd.sh.tpl
@@ -30,7 +30,7 @@ fi

# the general form of variable PEER_SERVICE_NAME is: "<clusterName>-pd-peer"
cluster_name=`echo ${PEER_SERVICE_NAME} | sed 's/-pd-peer//'`
domain="${HOSTNAME}.${PEER_SERVICE_NAME}.${NAMESPACE}.svc"
domain="${POD_NAME}.${PEER_SERVICE_NAME}.${NAMESPACE}.svc"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use POD_NAME instead of HOSTNAME environment. In Kubernetes, the hostname in the pod is implicitly set to the host hostname when the pod is running in host network (xref: kubernetes/kubernetes#12893). Currently, there is no way to change this behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POD_NAME is undefined if using old tidb-operator. To be compatible with old tidb-operator, set POD_NAME to $(hostname) or HOSTNAME if POD_NAME is undefined.

@weekface
Copy link
Contributor

Add a case in stability test?

@cofyc
Copy link
Contributor Author

cofyc commented Aug 15, 2019

Sure, I'll add a stability test.

@cofyc
Copy link
Contributor Author

cofyc commented Aug 20, 2019

I removed the needs-cherry-pick-1.0 label because it's a new feature, not a bugfix or improvement. We can target 1.1 release.

@cofyc
Copy link
Contributor Author

cofyc commented Aug 20, 2019

The stability test is under development, I'll update this PR soon.

@cofyc
Copy link
Contributor Author

cofyc commented Aug 20, 2019

/run-e2e-in-kind

@cofyc
Copy link
Contributor Author

cofyc commented Aug 20, 2019

/run-e2e-in-kind

@cofyc cofyc changed the title [WIP] Support hostNetwork Support hostNetwork Aug 20, 2019
@cofyc
Copy link
Contributor Author

cofyc commented Aug 20, 2019 •
edited
Loading

I've added a test in e2e. @weekface @tennix @xiaojingchen @gregwebs PTAL.
It's safe to switch back and forth between pod network and host network (if no port conflicts) because in Kubernetes host network can communicate with pod network. I also manually tested in GKE.

@cofyc
Copy link
Contributor Author

cofyc commented Aug 21, 2019

/run-e2e-in-kind

tennix
tennix approved these changes Aug 21, 2019
View reviewed changes
Copy link
Member

@tennix tennix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tennix
Copy link
Member

tennix commented Aug 21, 2019

In this mode, the pod should be set with anti-affinity to disable two pods scheduled on the same node. This should be added to dev document.

@cofyc cofyc merged commit e17d281 into pingcap:master Aug 21, 2019
@sre-bot sre-bot mentioned this pull request Aug 21, 2019
Closed
@sre-bot
Copy link
Contributor

sre-bot commented Aug 21, 2019

cherry pick to release-1.0 in PR #798

@cofyc cofyc mentioned this pull request Nov 20, 2019
Closed
cofyc added a commit to cofyc/tidb-operator that referenced this pull request Nov 20, 2019
@cofyc
Support hostNetwork (pingcap#774)
e2853ec 
* Support hostNetwork

* Use POD_NAME instead of HOSTNAME

* Update chart

* e2e test

* remove PodSpec
cofyc added a commit to cofyc/tidb-operator that referenced this pull request Nov 20, 2019
@cofyc
Support hostNetwork (pingcap#774)
6757338 
* Support hostNetwork

* Use POD_NAME instead of HOSTNAME

* Update chart

* e2e test

* remove PodSpec
cofyc added a commit that referenced this pull request Nov 21, 2019
@cofyc
WIP: Cherry-pick hostnetwork support (#774) to release-1.0 (#1202)
6b7bd19 
* Support hostNetwork (#774)

* Support hostNetwork

* Use POD_NAME instead of HOSTNAME

* Update chart

* e2e test

* remove PodSpec

* Use HOSTNAME if POD_NAME is unset for backward compatibility. (#818)

* fix bug (#1032)

* remove tests

* add POD_NAME env conditionally
yahonda pushed a commit that referenced this pull request Dec 27, 2021
@ran-huang
.github: add 2 exempt issue labels (#774)
1f9c822 
* .github: add 2 exempt issue labels

Signed-off-by: Ran <huangran@pingcap.com>

* Update .github/workflows/stale.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tennix tennix tennix approved these changes

@weekface weekface weekface approved these changes

@xiaojingchen xiaojingchen Awaiting requested review from xiaojingchen

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Able to run TiDB components in host network
4 participants
@cofyc @weekface @tennix @sre-bot