Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support of Azure standard load balancer and public IP #61884

Merged
merged 2 commits into from
Mar 30, 2018

Conversation

feiskyer
Copy link
Member

@feiskyer feiskyer commented Mar 29, 2018

What this PR does / why we need it:

This PR adds a new configure loadBalancerSku and excludeMasterFromStandardLB to support Azure standard load balancer and public IP.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
First step of #60485

Special notes for your reviewer:

This is the first step of #60485, and only VMAS is supported yet. VMSS support will be added in a following PR.

Release note:

Azure cloud provider now supports standard SKU load balancer and public IP. To use it, set cloud provider config with
{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, it will be default to true, which means master nodes are excluded to the backend of standard LB.

Also note standard load balancer doesn't work with annotation `service.beta.kubernetes.io/azure-load-balancer-mode`. This is because all nodes (except master) are added as the LB backends.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 29, 2018
@feiskyer feiskyer requested review from karataliu and brendandburns and removed request for karataliu March 29, 2018 09:16
@feiskyer
Copy link
Member Author

/sig azure

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 29, 2018
@feiskyer
Copy link
Member Author

/retest

matches := backendPoolIDRE.FindStringSubmatch(backendPool)
if len(matches) == 2 {
lbName := matches[1]
if strings.HasSuffix(lbName, InternalLoadBalancerNameSuffix) == isInternal {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible to pull master nodes into the pool, if it only has public/internal LB.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check before calling ensureHostInPool

@feiskyer
Copy link
Member Author

@karataliu As discussed offline, added two configs now. PTAL

{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, master nodes are also added to the backend of standard LB. This is useful for small clusters, e.g. 3 nodes with HA, in which case master nodes are also running workload Pods.

LoadBalancerSku string `json:"loadBalancerSku" yaml:"loadBalancerSku"`
// ExcludeMasterFromStandardLB excludes master nodes from standard load balancer.
ExcludeMasterFromStandardLB bool `json:"excludeMasterFromStandardLB" yaml:"excludeMasterFromStandardLB"`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should assign default values to those 2 somewhere. ExcludeMasterFromStandardLB should default to true?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExcludeMasterFromStandardLB is bool, so it is default to false. This works for all deployments

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using pointers thus it'll be easy to change default value

ref: Azure/acs-engine#1584 (review)

@@ -242,3 +243,11 @@ func (az *Cloud) newRouteTableCache() (*timedCache, error) {

return newTimedcache(rtCacheTTL, getter)
}

func (az *Cloud) useStandardLoadBalancer() bool {
if strings.ToLower(az.LoadBalancerSku) == loadBalancerSkuStandard {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can return strings.EqualFold(az.LoadBalancerSku,loadBalancerSkuStandard) directly

@@ -123,7 +130,7 @@ func (az *Cloud) mapLoadBalancerNameToVMSet(lbName string, clusterName string) (
// This would be the name for Azure LoadBalancer resource.
func (az *Cloud) getLoadBalancerName(clusterName string, vmSetName string, isInternal bool) string {
lbNamePrefix := vmSetName
if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) {
if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) || az.useStandardLoadBalancer() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anywhere to add hint that selectLoadBalancer does not work for StandardLoadBalancer?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, will add this check in getLoadBalancer()

},
{
description: "non-default standard external LB should map to empty string",
lbName: "azuretest-internal",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

external LB ?

}

for _, c := range cases {
if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should set LoadBalancerSku back for !useStandardLB. This works now since the Standard SKU cases are in the end.

for _, c := range cases {
if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, switch back for !useStandardLB

if c.useStandardLB {
az.Config.LoadBalancerSku = loadBalancerSkuStandard
}
vmset := az.getLoadBalancerName(c.clusterName, c.vmSet, c.isInternal)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better 'loadbalancerName := '

@feiskyer
Copy link
Member Author

@karataliu addressed comments. PTAL

@feiskyer
Copy link
Member Author

/retest

@@ -273,6 +284,11 @@ func (az *Cloud) selectLoadBalancer(clusterName string, service *v1.Service, exi
Location: &az.Location,
LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{},
}
if az.useStandardLoadBalancer() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not be true now due to ' wantLb && !az.useStandardLoadBalancer()'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack

@@ -623,6 +630,22 @@ func (as *availabilitySet) ensureHostInPool(serviceName string, nodeName types.N
}
}
if !foundPool {
if as.useStandardLoadBalancer() && len(newBackendPools) > 0 {
// Although standard load balancer supports backends from multiple availability
// sets, the same interface couldn't reference more than one load balancer of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to use full name 'interface' -> 'network interface', i got a bit confused when looking at the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so are you referring go interface? that make sense

@karataliu
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, karataliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants