Lipschitz continuity – Serlo

Lipschitz continuity is an even stronger form of uniform continuity and is often used in the theory of differential equations.

Derivation

By introducing the notion of Lipschitz continuity, we will be able to say more about the rate at which a function changes. As we already know, continuous functions have the neat property that sufficiently small changes in the arguments will result in arbitrary small changes in the function value. On top of that, Lipschitz continuity allows us give an estimate of that change. We can thus talk very precisely about how "fast" the changes of the output become smaller. For better understanding, let us first recap what we mean by changes of a function. Let $f:D\to \mathbb {R}$ be an function with domain $D$.

Let's take two arbitrary points ${\tilde {x}}$ and $x$ from the domain of $f$ and form the straight line through the points $({\tilde {x}},f({\tilde {x}}))$ und $(x,f(x))$. The slope of the line increases, as the difference between the values $f({\tilde {x}})$ and $f(x)$ gets bigger. The mean rate of change between function values is equal to the slope ${\textstyle {\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}}$ of the secant through those points and can be calculated with the so-called slope triangle:

Steigungsdreieck zum Bestimmen der Sekantensteigung

Let's assume now that the rate of change is bounded, i.e. the slopes of the secants cannot get arbitrary large or small. Thus the absolute value ${\textstyle \left|{\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}\right|}$ has an upper bound (because the absolute value is bounded, both positive and negative values are bounded). There exists a constant $L\in \mathbb {R}$, so that for all ${\tilde {x}},x\in D$ with $x\neq {\tilde {x}}$ the inequality ${\textstyle \left|{\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}\right|\leq L}$ holds: This number $L$ is called the Lipschitz constant. By multiplying both sides of the equation with $|x-{\tilde {x}}|$, we get:

${\begin{aligned}&&\left|{\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}\right|&\leq L\\[0.3em]&\iff &\left|f(x)-f({\tilde {x}})\right|&\leq L|x-{\tilde {x}}|\end{aligned}}$

This inequality $\left|f(x)-f({\tilde {x}})\right|\leq L|x-{\tilde {x}}|$ is the basis for the definition of Lipschitz continuity. If we can find such a constant $L$ that satisfies this inequality for all $x,{\tilde {x}}\in R$, then we have also found an absolute bound for the rate of change of the function. Because the inequality $\left|f(x)-f({\tilde {x}})\right|\leq L|x-{\tilde {x}}|$ is still true even for $x={\tilde {x}}$ this will result in a nice simplification of the definition, because we can drop the assumption that $x\neq {\tilde {x}}$, which we otherwise would have needed, had we tried to work with the original mean rate of change ${\textstyle {\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}}$.

Definition

Definition (Lipschitz continuity)

Let $D\subseteq \mathbb {R}$ be a subset of $\mathbb {R}$ and let $f:D\rightarrow \mathbb {R}$ be a function. Then we say $f$ is Lipschitz continuous in $D$, if and only if there exists a constant $L\geq 0$, which satisfies $|f(x)-f({\tilde {x}})|\leq L\cdot |x-{\tilde {x}}|$ for all $x,{\tilde {x}}\in D$.

More formally the definition could be written using quantifiers:

${\begin{array}{cc}&f:D\rightarrow \mathbb {R} {\text{ is Lipschitz continuous }}\\[1em]\iff &\exists L\geq 0\,\forall x,{\tilde {x}}\in D:|f(x)-f({\tilde {x}})|\leq L\cdot |x-{\tilde {x}}|\end{array}}$

The right-hand side of the above equivalence can be translated as follows:

$\underbrace {\exists L\geq 0} _{{\text{There exists }}L\geq 0}\underbrace {\forall x,{\tilde {x}}\in D:} _{{\text{, so that for all }}x,{\tilde {x}}{\text{ following is true: }}}\underbrace {|f(x)-f({\tilde {x}})|\leq L\cdot |x-{\tilde {x}}|} _{|f(x)-f({\tilde {x}})|{\text{ is smaller than }}L\cdot |x-{\tilde {x}}|}$

Why do we need Lipschitz continuity?

The Lipschitz constant of a Lipschitz continuous functions gives us an absolute upper bound for the rate of change. This is useful if we want to estimate function values.

Let us assume we are given a point $x\in D$ from the domain and the corresponding function value $f(x)$. Now imagine you want to estimate the value $f(y)$ for a new point $y\in D$. We can obtain such an estimate if we use our Lipschitz constant to find an upper and lower bound for $f(y)$. From Lipschitz continuity it follows that:

$\left|f(y)-f(x)\right|\leq L\left|y-x\right|\iff -L\left|y-x\right|\leq f(y)-f(x)\leq L\left|y-x\right|$

By adding $f(x)$ we obtain:

$f(x)-L|y-x|\leq f(y)\leq f(x)+L|y-x|$

This way we have found a lower and upper bound and know that $f(y)$ must be located somewhere in between.

Intuition

Picturing Lipschitz continuity with cones

Let's try to visualize the Lipschitz condition: Given a Lipschitz continuous function $f:D\to \mathbb {R}$ we draw the straight lines $g_{1}$ and $g_{2}$ through the point $({\tilde {x}},f({\tilde {x}}))$ and make their slope be equal to our Lipschitz constant $L$ and $-L$ respectively. Above we already have seen that the Lipschitz condition is given by ${\textstyle {\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}\leq L}$, which means that the slope of all secants through $({\tilde {x}},f({\tilde {x}}))$ is bounded. This means that the graph of the function must run between those two lines:

Visualisierung Lipschitz

This constraints is true for all ${\tilde {x}}$ in the domain. In the picture we could let the lines move along the graph of $f$ and in turn the graph of $f$ will always lie between those two lines:

Lipschitz Animation

If the function is differentiable the derivative of the function taken at a point will match the slope of the tangent at that point. Using this visualization we can see that the derivative of a Lipschitz continuous function cannot get bigger than $L$ (or less than $-L$). This means that the absolute value of the derivative is bounded by the Lipschitz constant.

Difference between Lipschitz and uniform continuity

With uniform continuous functions to every $\epsilon >0$ we can find a $\delta >0$, so that the function values lie in the $\epsilon$-$\delta$-rectangle. But this rectangle only allows for a worse estimation compared to the Lipschitz inequality.

Both the Lipschitz and uniform continuity are so-called global properties of a function (normal continuity is only a local property!). In contrast to Lipschitz continuity, uniform continuity gives no information about the rate of change. This could be a disadvantage if you are trying to estimate a value $f(y)$ from another value $f(x)$ .

Lipschitz continuity implies a linear relationship between the distance of the arguments $y-x$ and the output values $f(y)-f(x)$. This is not the case with uniform continuity because the constraint $\left|f(y)-f(x)\right|\leq \epsilon$ is the same for all $x,y$ with $\left|y-x\right|\leq \delta$. Hence the cones of Lipschitz continuity that become progressively smaller as we approach $x$ give us a much better estimate than the „$\epsilon$-$\delta$-rectangle“ of uniform continuity. We could try choose $\epsilon$ arbitrary small, to improve the approximation of $f(y)$. But this could mean that our $\delta$ values must become really small too, i.e. we have to be really close the the original point $x$. You can observe this phenomenon by studying the behaviour of the square-root function near $x=0$.

Link to uniform continuity

How is this new notion of Lipschitz continuity linked to the old concepts of normal and uniform continuity? We already made the claim that Lipschitz continuity is stronger than uniform continuity: Every Lipschitz continuous function is also uniform continuos. But the converse is not generally true.

Lipschitz continuous functions are uniformly continuous

Theorem (Lipschitz continuity implies uniform continuity)

Let $D\subseteq \mathbb {R}$ and $f:D\rightarrow \mathbb {R}$ be a function which is Lipschitz continuous in $D$. Then $f$ is also uniformly continuous in $D$.

Proof (Lipschitz continuity implies uniform continuity)

Let's recall the definition of uniform continuity: A function $f$ is uniformly continuous, if and only if for all $\epsilon >0$ we can find a $\delta >0$, so that for all $x,y\in D$ with $|x-y|<\delta$, it follows that $|f(x)-f(y)|<\epsilon$.

Thus let $\epsilon >0$. Our job is to find a $\delta >0$, so that for all arguments $x,y\in D$ that are less than $\delta$ apart from one another, i.e. $|x-y|<\delta$, also the distance of the outputs is smaller than $\epsilon$, so $|f(x)-f(y)|<\epsilon$ should be satisfied. Our starting point is that $f$ is Lipschitz continuous, and we are trying to show that $f$ must be uniformly continuous also. We remember that being Lipschitz continuous means, that there exists a constant $L\geq 0$, so that $|f(x)-f(y)|\leq L\cdot |x-y|$ for all $x,y\in D$.

The question we need to ask ourself is: How large can the distance between the arguments $x,y\in D$ be, so that the outputs still satisfy $|f(x)-f(y)|<\epsilon$? Maybe you will soon realize that we can choose the distance $|x-y|$ to be smaller than ${\tfrac {\epsilon }{L}}$. So we choose our $\delta ={\tfrac {\epsilon }{L}}$. But be wary! The Lipschitz constant $L$ could be zero (we would divide by zero). Here we need a case-by-case analysis:

Fall 1: $L=0$

If $L=0$, then the Lipschitz condition tells us that $|f(x)-f(y)|\leq 0$ for all $x,y\in D$. But this implies that $|f(x)-f(y)|=0$ and thus $f(x)=f(y)$ for all $x,y\in D$. That means $f$ must be a constant function, and is therefore uniformly continuous. For an arbitrary $\epsilon >0$ you could choose any $\delta >0$.

Fall 2: $L\neq 0$

The interesting case is $L\neq 0$. We already hinted at the solution. Choose $\delta ={\tfrac {\epsilon }{L}}$. Let $x,y\in D$ with $|x-y|\leq \delta$. Now everything will follow from our Lipschitz condition:

${\begin{aligned}|f(x)-f(y)|&\leq L\cdot |x-y|<L\cdot \delta \\[0.5em]&=L\cdot {\frac {\epsilon }{L}}=\epsilon \end{aligned}}$

Not all uniformly continuous functions are Lipschitz continuous

To show this fact we only need to find a counterexample, i.e. a function which we know is uniformly continuous, but not Lipschitz continuous. We will see that the square root function in $\mathbb {R} _{0}^{+}$ is such a counterexample. It was already proven previously that this function is uniformly continuous. We now will show that it is not Lipschitz continuous. Thus consider the function:

${\begin{aligned}f:\mathbb {R} _{0}^{+}&\to \mathbb {R} \\x&\mapsto {\sqrt {x}}\end{aligned}}$

We will give a prove by contradiction. Let's assume this function is Lipschitz continuous. By definition there must be a constant $L\geq 0$, so that for all $x,y\in \mathbb {R} _{0}^{+}$ the following holds: $|{\sqrt {x}}-{\sqrt {y}}|<L\cdot |x-y|$. But this would have further implications for all $x,y\in \mathbb {R} _{0}^{+}$ with $x>0$ or $y>0$:

${\begin{aligned}&&|{\sqrt {x}}-{\sqrt {y}}|&<L\cdot |x-y|\\[0.3em]&\implies &{\frac {|{\sqrt {x}}-{\sqrt {y}}|\cdot |{\sqrt {x}}+{\sqrt {y}}|}{|{\sqrt {x}}+{\sqrt {y}}|}}&<L\cdot |x-y|\\[0.3em]&\implies &{\frac {1}{|{\sqrt {x}}+{\sqrt {y}}|}}\cdot |x-y|&<L\cdot |x-y|\\[0.3em]&\implies &{\frac {1}{|{\sqrt {x}}+{\sqrt {y}}|}}&<L\end{aligned}}$

But by choosing ${\textstyle x={\frac {1}{4L^{2}}}>0}$ and $y=0$, we see that:

${\begin{aligned}{\frac {1}{|{\sqrt {x}}+{\sqrt {y}}|}}&={\frac {1}{{\frac {1}{\sqrt {4L^{2}}}}+0}}\\[0.5em]&={\frac {1}{\frac {1}{2L}}}=2L>L\end{aligned}}$

This is a contradiction! Therefore the assumption that $f$ is Lipschitz continuous in $\mathbb {R} _{0}^{+}$ must be false.

Lipschitz continuity implies (normal) continuity

From the previous section we see that Lipschitz continuity implies (normal) continuity: Since Lipschitz continuous functions are uniformly continuous, they are also continuous (remember that uniform continuity is stronger than normal continuity). To view it from a different angle, we can ask ourself why a discontinuous function cannot be Lipschitz continuous:

Let's re-evoke the first intuitive image we had about continuity: A function is continuous if there aren't any "jumps" or "gaps":

Sprungstelle

The pictured function is discontinuous at ${\tilde {x}}=1$. Imagine tracing a straight line through the point at the ${\tilde {x}}$-value of the graph, and through another point $x>{\tilde {x}}$. This line is a secant of that graph.

Sekante an der Sprungstelle

If you imagine moving the right intersection point of that secant closer and closer to the point $x$ which is the jump point, you will observe that the slope of the secant gets bigger and bigger and will eventually reach infinity. This means that there cannot be any constant $L\in \mathbb {R}$ that will bound the absolute value of the slope of the secant. If you think you have found such a constant $L$ just move the intersection point of the secant even closer to $x$, until you will eventually reach a point where the slope ${\textstyle {\frac {f(x)-f({\tilde {x}})}{x-{\tilde {x}}}}}$ will be greater than $L$.

So now it should be clear why every discontinuous function cannot be Lipschitz continuous. You can reverse this proposition logically by contraposition and obtain the original proposition that Lipschitz continuity implies (normal) continuity. In summary, functions which rate of change is bounded are Lipschitz continuous and cannot have any points of discontinuity.

Sekante einer Funktion ohne Sprungstelle

Examples

Example (The identity is Lipschitz continuous)

The identity $f:\mathbb {R} \to \mathbb {R}$ with $f(x)=x$ is Lipschitz continuous on all of $\mathbb {R}$ with Lipschitz constant $L=1$, because for all $x,y\in \mathbb {R}$ it follows that:

$|f(x)-f(y)|=|x-y|\leq 1\cdot |x-y|$

Example (The restricted Square function is Lipschitz continuous)

Consider the Square function on the closed interval $[a,b]$:

${\begin{aligned}f:[a,b]&\to \mathbb {R} \\x&\mapsto x^{2}\end{aligned}}$

The values $a,b\in \mathbb {R}$ are chosen at will, but with $a<b$. We claim that this function is Lipschitz continuous because for all $x,y\in [a,b]$ it follows that:

${\begin{aligned}|f(x)-f(y)|&=|x^{2}-y^{2}|\\[0.3em]&=|(x+y)\cdot (x-y)|\\[0.3em]&=|x+y|\cdot |x-y|\\[0.3em]&\leq \left(\left|x\right|+\left|y\right|\right)\cdot \left|x-y\right|\\[0.3em]&\leq 2\cdot \max\{\left|x\right|,\left|y\right|\}\cdot \left|x-y\right|\\[0.3em]&\leq 2\cdot \max\{|a|,|b|\}\cdot |x-y|\end{aligned}}$

Thus we can choose $L=2\cdot \max\{|a|,|b|\}$ as our Lipschitz constant.

Example (The (unrestricted) Square function is not Lipschitz continuous)

In contrast consider the unrestricted Square function $f:\mathbb {R} \to \mathbb {R}$ with $f(x)=x^{2}$. By dropping the condition that the interval must be closed, we lose the Lipschitz continuity. Assume that there is a Lipschitz constant $L>0$ with $|f(x)-f(y)|\leq L\cdot |x-y|$ for all $x,y\in \mathbb {R}$. We start by observing that:

${\begin{aligned}|f(x)-f(y)|&=|x^{2}-y^{2}|\\&=|(x+y)(x-y)|\\&=|x+y|\cdot |x-y|\end{aligned}}$

Because of $|f(x)-f(y)|\leq L\cdot |x-y|$ the inequality $|x+y|\leq L$ must be true for all $x,y\in \mathbb {R}$. But it's clear that such a Lipschitz constant cannot possibly exist. We easily find that by setting $x=0$ and $y=L+1$ (after trying to choose a fixed $L$) the inequality doesn't hold. By this contradiction we see that the Square function $f$ cannot be Lipschitz continuous on all of $\mathbb {R}$

Exercises

Lineare Funktionen sind Lipschitz-stetig

Exercise (Linear functions are Lipschitz continuous)

Show that every linear function $f:\mathbb {R} \to \mathbb {R} :x\mapsto a\cdot x+b$ with $a,b\in \mathbb {R}$ is Lipschitz continuous.

Solution (Linear functions are Lipschitz continuous)

Let $x,y\in \mathbb {R}$ be at will. Then we see:

${\begin{aligned}|f(x)-f(y)|&=|a\cdot x+b-(a\cdot y+b)|\\[0.3em]&=|a\cdot (x-y)|=|a|\cdot |x-y|\end{aligned}}$

The linear function $f$ is therefore Lipschitz continuous with Lipschitz constant $L=|a|$.

Quadratic functions and Lipschitz continuity

Exercise (Quadratic functions and Lipschitz continuity)

Prove that every quadratic function $f:\mathbb {R} \to \mathbb {R}$ with $f(x)=a\cdot x^{2}+b\cdot x+c$ for $a,b,c\in \mathbb {R}$ and $a\neq 0$ Is Lipschitz continuous on the compact interval $[-2,2]\subseteq \mathbb {R}$ , but not on $\mathbb {R}$ .

Solution (Quadratic functions and Lipschitz continuity)

We use a binomial expansion:

${\begin{aligned}|f(x)-f(y)|&=|ax^{2}+bx+c-(ay^{2}+by+c)|\\[0.3em]&=|a(x^{2}-y^{2})+b(x-y)|\\[0.3em]&=|a(x+y)(x-y)+b(x-y)|\\[0.3em]&=|a(x+y)+b|\cdot |x-y|\end{aligned}}$

Let now $x,y\in [-2,2]$. The triangle inequality implies:

${\begin{aligned}|f(x)-f(y)|&=|a(x+y)+b|\cdot |x-y|\\[0.3em]&\leq (|a(x+y)|+|b|)\cdot |x-y|\\[0.3em]&\leq (|a|\cdot |x+y|+|b|)\cdot |x-y|\\[0.3em]&\leq (|a|\cdot (|x|+|y|)+|b|)\cdot |x-y|\\[0.3em]&\leq (4|a|+|b|)\cdot |x-y|\end{aligned}}$

So the graph of the function can never be steeper than $L=(4|a|+|b|)$ and this $L$ is a suitable Lipschitz constant.

But what happens if we take the entire real axis as domain $D=\mathbb {R}$ ? The function is getting infinitely steep as $x$ runs towards $\infty$ so we expect that it is not Lipschitz-continuous, i.e. there is no upper bound $L$ for the steepness. Assume, $f$ was Lipschitz continuous with upper bound on the steepness $L\geq 0$ , $|f(x)-f(y)|\leq L\cdot |x-y|$ for all $x,y\in \mathbb {R}$ The computation above yields

${\begin{aligned}|f(x)-f(y)|&=|a(x+y)+b|\cdot |x-y|\\[0.3em]&\leq L\cdot |x-y|\end{aligned}}$

So the steepness should be at most $|a(x+y)+b|\leq L$ for all $x,y\in \mathbb {R}$ . But if we choose $y=0$ and ${\textstyle x={\frac {L-b+1}{a}}\in \mathbb {R} }$, then we exceed the maximum steepness:

${\begin{aligned}|a(x+y)+b|&=\left|a\left({\frac {L-b+1}{a}}+0\right)+b\right|\\[0.3em]&=|L+1|>L\end{aligned}}$

This is a contradiction and hence, $f$ is not Lipschitz continuous on $\mathbb {R}$.

Lipschitz-Continuity and the hyperbola

Exercise

Let $f:(0,\infty )\to \mathbb {R}$ with ${\textstyle f(x)={\frac {1}{x}}}$.

1. Prove that $f|_{[1,\infty )}$ is Lipschitz continuous and determine the Lipschitz constant for this function (on this interval).
2. Is the function still Lipschitz cotinuous if we extend its domain of definition to $(0,1]$ ?

Solution

Solution sub-exercise 1:

Let $x,y\in [1,\infty )$. Then,

${\begin{aligned}|f(x)-f(y)|&=\left|{\frac {1}{x}}-{\frac {1}{y}}\right|=\left|{\frac {y-x}{xy}}\right|\\[0.3em]&={\frac {1}{xy}}\cdot |x-y|\\[0.3em]&{\color {OliveGreen}\left\downarrow \ x,y\geq 1\right.}\\[0.3em]&\leq 1\cdot |x-y|\end{aligned}}$

So $f$ is Lipschitz-continuous on $[1,\infty )$ with Lipschitz constant $L=1$ (= maximal steepness). There is no better constant, since the first derivative at 1 is $|f'(1)|=1$ , so the function has steepness 1 there.

Solution sub-exercise 2:

Assume, $f|_{(0,1]}$ would be Lipschitz continuous . Then there is a Lipschitz constant $L\geq 0$ with ${\textstyle |f(x)-f(y)|={\frac {1}{xy}}|x-y|\leq L|x-y|}$ for all $x,y\in (0,1]$. For $x\neq y$ this is equivalent to ${\textstyle {\frac {1}{xy}}\leq L}$. Now, we distinguish the two cases ${\textstyle L>{\frac {1}{2}}}$ and ${\textstyle L\leq {\frac {1}{2}}}$:

Fall 1: ${\textstyle L>{\frac {1}{2}}}$

For ${\textstyle x={\frac {1}{2L}}\in (0,1]}$ and $y=1$ there is ${\textstyle {\frac {1}{xy}}=2L>L}$. ↯

Fall 2: ${\textstyle L\leq {\frac {1}{2}}}$

For $x=2L\in (0,1]$ and $y=1$ we get a contradiction:

${\frac {1}{xy}}={\frac {1}{2L}}\ {\overset {L\leq {\frac {1}{2}}}{\geq }}\ {\frac {1}{2\cdot {\frac {1}{2}}}}=1>L$

A useful criterion for Lipschitz continuity

The Mean value theorem can be used to show that on a compact interval, every continuous differentiable function is Lipschitz continuous:

Theorem (Lipschitz continuity of differentiable functions on compact intervals)

Let $f:[a,b]\to \mathbb {R}$ be continuous and further differentiable in $(a,b)$ . The derivative function $f':(a,b)\to \mathbb {R}$ is assumed to be bounded (this is especially the case, if $f'$ is continuous). Then, $f$ is Lipschitz continuous with Lipschitz constant $L=\sup _{x\in (a,b)}|f'(x)|$.

The derivative being bounded by $|f'(x)|\leq L$ intuitively corresponds to having a maximum steepness, which in turn implies Lipschitz continuity. However, there are non-differentiable functions with maximum steepness, so Lipschitz continuity actually requires less than being differentiable and classifies more functions.