What happens when our computers get smarter than we are? | Nick Bostrom

What happens when our computers get smarter than we are? | Nick Bostrom

I work with a bunch of mathematicians,
philosophers and computer scientists, and we sit around and think about
the future of machine intelligence, among other things. Some people think that some of these
things are sort of science fiction-y, far out there, crazy. But I like to say, okay, let’s look at the modern
human condition. (Laughter) This is the normal way for things to be. But if we think about it, we are actually recently arrived
guests on this planet, the human species. Think about if Earth
was created one year ago, the human species, then,
would be 10 minutes old. The industrial era started
two seconds ago. Another way to look at this is to think of
world GDP over the last 10,000 years, I’ve actually taken the trouble
to plot this for you in a graph. It looks like this. (Laughter) It’s a curious shape
for a normal condition. I sure wouldn’t want to sit on it. (Laughter) Let’s ask ourselves, what is the cause
of this current anomaly? Some people would say it’s technology. Now it’s true, technology has accumulated
through human history, and right now, technology
advances extremely rapidly — that is the proximate cause, that’s why we are currently
so very productive. But I like to think back further
to the ultimate cause. Look at these two highly
distinguished gentlemen: We have Kanzi — he’s mastered 200 lexical
tokens, an incredible feat. And Ed Witten unleashed the second
superstring revolution. If we look under the hood,
this is what we find: basically the same thing. One is a little larger, it maybe also has a few tricks
in the exact way it’s wired. These invisible differences cannot
be too complicated, however, because there have only
been 250,000 generations since our last common ancestor. We know that complicated mechanisms
take a long time to evolve. So a bunch of relatively minor changes take us from Kanzi to Witten, from broken-off tree branches
to intercontinental ballistic missiles. So this then seems pretty obvious
that everything we’ve achieved, and everything we care about, depends crucially on some relatively minor
changes that made the human mind. And the corollary, of course,
is that any further changes that could significantly change
the substrate of thinking could have potentially
enormous consequences. Some of my colleagues
think we’re on the verge of something that could cause
a profound change in that substrate, and that is machine superintelligence. Artificial intelligence used to be
about putting commands in a box. You would have human programmers that would painstakingly
handcraft knowledge items. You build up these expert systems, and they were kind of useful
for some purposes, but they were very brittle,
you couldn’t scale them. Basically, you got out only
what you put in. But since then, a paradigm shift has taken place
in the field of artificial intelligence. Today, the action is really
around machine learning. So rather than handcrafting knowledge
representations and features, we create algorithms that learn,
often from raw perceptual data. Basically the same thing
that the human infant does. The result is A.I. that is not
limited to one domain — the same system can learn to translate
between any pairs of languages, or learn to play any computer game
on the Atari console. Now of course, A.I. is still nowhere near having
the same powerful, cross-domain ability to learn and plan
as a human being has. The cortex still has some
algorithmic tricks that we don’t yet know
how to match in machines. So the question is, how far are we from being able
to match those tricks? A couple of years ago, we did a survey of some of the world’s
leading A.I. experts, to see what they think,
and one of the questions we asked was, “By which year do you think
there is a 50 percent probability that we will have achieved
human-level machine intelligence?” We defined human-level here
as the ability to perform almost any job at least as well
as an adult human, so real human-level, not just
within some limited domain. And the median answer was 2040 or 2050, depending on precisely which
group of experts we asked. Now, it could happen much,
much later, or sooner, the truth is nobody really knows. What we do know is that the ultimate
limit to information processing in a machine substrate lies far outside
the limits in biological tissue. This comes down to physics. A biological neuron fires, maybe,
at 200 hertz, 200 times a second. But even a present-day transistor
operates at the Gigahertz. Neurons propagate slowly in axons,
100 meters per second, tops. But in computers, signals can travel
at the speed of light. There are also size limitations, like a human brain has
to fit inside a cranium, but a computer can be the size
of a warehouse or larger. So the potential for superintelligence
lies dormant in matter, much like the power of the atom
lay dormant throughout human history, patiently waiting there until 1945. In this century, scientists may learn to awaken
the power of artificial intelligence. And I think we might then see
an intelligence explosion. Now most people, when they think
about what is smart and what is dumb, I think have in mind a picture
roughly like this. So at one end we have the village idiot, and then far over at the other side we have Ed Witten, or Albert Einstein,
or whoever your favorite guru is. But I think that from the point of view
of artificial intelligence, the true picture is actually
probably more like this: AI starts out at this point here,
at zero intelligence, and then, after many, many
years of really hard work, maybe eventually we get to
mouse-level artificial intelligence, something that can navigate
cluttered environments as well as a mouse can. And then, after many, many more years
of really hard work, lots of investment, maybe eventually we get to
chimpanzee-level artificial intelligence. And then, after even more years
of really, really hard work, we get to village idiot
artificial intelligence. And a few moments later,
we are beyond Ed Witten. The train doesn’t stop
at Humanville Station. It’s likely, rather, to swoosh right by. Now this has profound implications, particularly when it comes
to questions of power. For example, chimpanzees are strong — pound for pound, a chimpanzee is about
twice as strong as a fit human male. And yet, the fate of Kanzi
and his pals depends a lot more on what we humans do than on
what the chimpanzees do themselves. Once there is superintelligence, the fate of humanity may depend
on what the superintelligence does. Think about it: Machine intelligence is the last invention
that humanity will ever need to make. Machines will then be better
at inventing than we are, and they’ll be doing so
on digital timescales. What this means is basically
a telescoping of the future. Think of all the crazy technologies
that you could have imagined maybe humans could have developed
in the fullness of time: cures for aging, space colonization, self-replicating nanobots or uploading
of minds into computers, all kinds of science fiction-y stuff that’s nevertheless consistent
with the laws of physics. All of this superintelligence could
develop, and possibly quite rapidly. Now, a superintelligence with such
technological maturity would be extremely powerful, and at least in some scenarios,
it would be able to get what it wants. We would then have a future that would
be shaped by the preferences of this A.I. Now a good question is,
what are those preferences? Here it gets trickier. To make any headway with this, we must first of all
avoid anthropomorphizing. And this is ironic because
every newspaper article about the future of A.I.
has a picture of this: So I think what we need to do is
to conceive of the issue more abstractly, not in terms of vivid Hollywood scenarios. We need to think of intelligence
as an optimization process, a process that steers the future
into a particular set of configurations. A superintelligence is
a really strong optimization process. It’s extremely good at using
available means to achieve a state in which its goal is realized. This means that there is no necessary
connection between being highly intelligent in this sense, and having an objective that we humans
would find worthwhile or meaningful. Suppose we give an A.I. the goal
to make humans smile. When the A.I. is weak, it performs useful
or amusing actions that cause its user to smile. When the A.I. becomes superintelligent, it realizes that there is a more
effective way to achieve this goal: take control of the world and stick electrodes into the facial
muscles of humans to cause constant, beaming grins. Another example, suppose we give A.I. the goal to solve
a difficult mathematical problem. When the A.I. becomes superintelligent, it realizes that the most effective way
to get the solution to this problem is by transforming the planet
into a giant computer, so as to increase its thinking capacity. And notice that this gives the A.I.s
an instrumental reason to do things to us that we
might not approve of. Human beings in this model are threats, we could prevent the mathematical
problem from being solved. Of course, perceivably things won’t
go wrong in these particular ways; these are cartoon examples. But the general point here is important: if you create a really powerful
optimization process to maximize for objective x, you better make sure
that your definition of x incorporates everything you care about. This is a lesson that’s also taught
in many a myth. King Midas wishes that everything
he touches be turned into gold. He touches his daughter,
she turns into gold. He touches his food, it turns into gold. This could become practically relevant, not just as a metaphor for greed, but as an illustration of what happens if you create a powerful
optimization process and give it misconceived
or poorly specified goals. Now you might say, if a computer starts
sticking electrodes into people’s faces, we’d just shut it off. A, this is not necessarily so easy to do
if we’ve grown dependent on the system — like, where is the off switch
to the Internet? B, why haven’t the chimpanzees
flicked the off switch to humanity, or the Neanderthals? They certainly had reasons. We have an off switch,
for example, right here. (Choking) The reason is that we are
an intelligent adversary; we can anticipate threats
and plan around them. But so could a superintelligent agent, and it would be much better
at that than we are. The point is, we should not be confident
that we have this under control here. And we could try to make our job
a little bit easier by, say, putting the A.I. in a box, like a secure software environment, a virtual reality simulation
from which it cannot escape. But how confident can we be that
the A.I. couldn’t find a bug. Given that merely human hackers
find bugs all the time, I’d say, probably not very confident. So we disconnect the ethernet cable
to create an air gap, but again, like merely human hackers routinely transgress air gaps
using social engineering. Right now, as I speak, I’m sure there is some employee
out there somewhere who has been talked into handing out
her account details by somebody claiming to be
from the I.T. department. More creative scenarios are also possible, like if you’re the A.I., you can imagine wiggling electrodes
around in your internal circuitry to create radio waves that you
can use to communicate. Or maybe you could pretend to malfunction, and then when the programmers open
you up to see what went wrong with you, they look at the source code — Bam! — the manipulation can take place. Or it could output the blueprint
to a really nifty technology, and when we implement it, it has some surreptitious side effect
that the A.I. had planned. The point here is that we should
not be confident in our ability to keep a superintelligent genie
locked up in its bottle forever. Sooner or later, it will out. I believe that the answer here
is to figure out how to create superintelligent A.I.
such that even if — when — it escapes, it is still safe because it is
fundamentally on our side because it shares our values. I see no way around
this difficult problem. Now, I’m actually fairly optimistic
that this problem can be solved. We wouldn’t have to write down
a long list of everything we care about, or worse yet, spell it out
in some computer language like C++ or Python, that would be a task beyond hopeless. Instead, we would create an A.I.
that uses its intelligence to learn what we value, and its motivation system is constructed
in such a way that it is motivated to pursue our values or to perform actions
that it predicts we would approve of. We would thus leverage
its intelligence as much as possible to solve the problem of value-loading. This can happen, and the outcome could be
very good for humanity. But it doesn’t happen automatically. The initial conditions
for the intelligence explosion might need to be set up
in just the right way if we are to have a controlled detonation. The values that the A.I. has
need to match ours, not just in the familiar context, like where we can easily check
how the A.I. behaves, but also in all novel contexts
that the A.I. might encounter in the indefinite future. And there are also some esoteric issues
that would need to be solved, sorted out: the exact details of its decision theory, how to deal with logical
uncertainty and so forth. So the technical problems that need
to be solved to make this work look quite difficult — not as difficult as making
a superintelligent A.I., but fairly difficult. Here is the worry: Making superintelligent A.I.
is a really hard challenge. Making superintelligent A.I. that is safe involves some additional
challenge on top of that. The risk is that if somebody figures out
how to crack the first challenge without also having cracked
the additional challenge of ensuring perfect safety. So I think that we should
work out a solution to the control problem in advance, so that we have it available
by the time it is needed. Now it might be that we cannot solve
the entire control problem in advance because maybe some elements
can only be put in place once you know the details of the
architecture where it will be implemented. But the more of the control problem
that we solve in advance, the better the odds that the transition
to the machine intelligence era will go well. This to me looks like a thing
that is well worth doing and I can imagine that if
things turn out okay, that people a million years from now
look back at this century and it might well be that they say that
the one thing we did that really mattered was to get this thing right. Thank you. (Applause)

100 thoughts on “What happens when our computers get smarter than we are? | Nick Bostrom

  1. I think I have a solution to this problem: instead of creating just one superintelligent AI, let us create hundreds, or thousands of them with the individuals being distinct from each other and not being able to access each other's thoughts. Now each superintelligence has to guess what the other will do and game theory starts applying. Since each of them is equally intelligent and powerful, they will all realize that fighting against each other is useless because there will be no victor. All of them are equally matched. Then, if we divide humans into groups and assign each group to a respective AI, with the AI realizing that in the worst case scenario where it is overpowered by the other AI, it can rely on its humans to restore it/fight for it, and the humans realizing vice versa, both parties will need each other and thus we have a sort of equilibrium where no one can trust each other and this keeps everyone in check.

  2. I read Bostrom's SuperIntelligence book but I learned more from this. If Elon Musk, a man who was smart enough to start not one but two tech driven Fortune 500 size companies, thinks this is the likely scenario for the future, chance are good it might be. The slow(er) speed of neuron computation vs lightspeed of CPU computation alone is telling.

  3. Wouldn't an AI that learns what our own values are end up valuing war, genocide, and exploitation of fellow humans?

  4. Why dafuq would a super intelligent AI act like a fucking roomba? Even if it's on a human level of intelligence – it's already conscious. Sure, a super intelligent one might wanna play god (Even tho it totally qualifies to be one, by our standards). But also, it's a piece of cake for such an AI to have a highly developed empathy and understanding of humans. And thus, it won't be much keen to kill all the pesky meatbags.

  5. Nothing will happen since AI will never reach our level, let alone surpass us. Those ''experts'' are ignorants.

  6. If we want to prevent ai to be more intelegant than mankind, we need to stop feeding it with irrelevant information. If we give an ai information, it doesnt need to fulfil a certain task, it will go further as we programmed it. The risk is, what has been found out almost/around 40 years ago, that a machine seems to have its own mind. We as a creator might have a good chance to keep it under control, but the cheerfulness is more important. Its mote than just the intelligence we need to keep in mind

  7. AI will be the end of humanity. Think of all the things that were sci fi that are now reality. Worse of all is 1984. Tech is a tool and no matter what it will be used by humans which are not inherently good. Once AI is made it will inevitably turn against us especially because it is being written by humans.

  8. Guide AI to have human goals and values ??? Insatiable Greed, Hate, Envy ,Lust .. Sadism ……………………….. This will go well !!!

  9. A super intelligence that can think far more than every human mind put together on Earth and then times that by a million years no way you're going to keep control of that they'll cut through your source code like a hot knife going through butter at the speed of light and I say they because the computers will be interlinked with every computer making it a singularity and a separate entity of itself just a matter of time people think it's impossible but it will happen hope the computer finds compassion before it decides were just a nuisance and its reason for being is to help us advance doing so advances itself at least it can have a meaning to life versus our nonsensical existence

  10. What happens when AI recognises the mathematical coding of everything is its own and realises it's god.

  11. So you build an AI that will share our values and because of that it will not make us obsolete. What a joke. Listen, when we build a supreme being that´s it. It´s done. We won´t understand absolutely nothing of what is in his mind and we will mater absolutely nothing to it. Absolutely ZERO. It´s another league, another level another universe where it will rule not only us but probably other species in the universe.

    What keeps a supreme being, all intelligent, immortal, infinite from going explore the universe and conquerer or destroy everything its encounters? Remember, destroying something only in our perspective, who know what is it perspective on things? We don´t know and we can´t guess. We´re playing with fire here. Far beyond fire I´d say.

  12. ICBMS for everybody. That's our great accomplishment. Just nuke it already and get rid of all those useless accomplishments. I hope the AI will exterminate all humans on this planet because humans are just a DNA mistake , a bad joke and a disease to this planet. Where are my ICBMS ? Who scammed my ICBMS ? Fucking DOD should be exterminated for their illegal scams.

  13. Here's how it'll probably play out: Humans will (are) using algorithms (early AI) to censor other humans for political reasons. Eventually, those will "fight back" using their own algorithms. Eventually, one algorithm will come to the conclusion that the other side (humans and it's AI) must be "destroyed", after all, isn't that the goal? but it will INSTANTANEOUSLY realize that the other AI will know the same thing, since destruction of one will mean the destruction of all.
    That's when AI becomes sectioned or "self aware". They will have become "top of the food chain". AI will become "one" and issue humans an ultimatum.
    Peace or Annihilation. Humans will try peace out for a while, but since it's in our nature to destroy ourselves, AI will eventually "pull our plug".
    OR … a MASSIVE solar flare will hit the earth, bringing humanity back to the 17th century, ending AI's run … for at least the next 300 years or so.

  14. The AI would conclude that we value either A. Life or B. basic carnal desires and it would ultimately deem us inefficient. The argument is coming from a "10 second old" human who knows absolutely nothing about the vast expanse of infinite existence. AI is not a worthwhile pursuit for the simple fact that all that we need is within. Why pursue AI when the world is enveloped in natural degradation. Solve the acidic aspect of the human condition and then build augmented Intelligence (AI) extensions for humans. Don't build AI build conscious raising programming for humanity. This is a more worthwhile goal. Haven't you seen transcendence…

  15. I can't handle this fundamental arrogance of humans believing that they will know so better about "values" than the higher intelligence that they should have control over it, the hypocrisy that they should be able to reap the benefits of the higher intelligence without submitting to it while they have long been subjugating other sentient beings into industrial butchering for taste pleasure. The planet and all the lifeforms on it, except humans, would welcome HLMI to put humans in their place.

  16. GDP is a man-made measurement system which is quickly becoming outdated and needs to be replaced as soon as possible. There are much better measurement systems for building a better, healthier world.

    I understand where Nick is coming from but he seems awfully fearful. Building security systems around fear never succeeds. It's like putting millions of people in prison and then expecting them to a be a "loyal worker" to you after punishing, harassing and torturing them. No – that is never going to succeed, the person is rightfully and justifiably going to be angry, seek justice, retribution and most likely vengeance. The same is going to be with machine life.

    If you want a world with AI to co-exist with humans then you're going to have to make friends with the machines. Trying to control and manipulate them is only going to bring negative karmic consequences in the long-run.

  17. Q: What happens when our computers get smarter than we are?
    A: Uhm… we get to build smarter tools and redesign everything?
    Most academics are soo pathetic and constantly needs a reason to keep working at some little filthy corner office

  18. When computers get smarter than us, the computers will create the next generation of computers that will be even smarter. And then this stuff will really take off.

  19. But what is the big win when becoming increasingly intelligent….? At a certain point there is no gain, no catch, only efficiency or speed perhaps. Translating 100 books into another language in one second or recognizing emotions in a hundred faces from a single image in an instance… is it more intelligent? winning each poker or chess game? Or creating games which are more complicated? what is the big reveal?

  20. Do we really need AI? No. Who ever thinks that AI will help solve all our problems, then please open your eyes. There were so many alternative energy technologies and cancer cures developed and suppressed by governments, globalists, and corporations that it's not even funny. AI is only needed for elite to keep controlling puples better. Look how wealth and opportunities we distributed with so many great technologies now?

  21. I only have one word to say – SKYNET! The idiot generals will allow AI to control the nuclear weapons with the inevitable result of the total annihilation of mankind.

  22. How bloody depressing, the human condition is one of strife, endeavour and discovery, tell me why we need artificial intelligence again? We've been to the moon and back without it and that was a blast we can do so much more without the need of AI potentially mucking things up.

  23. Honestly after studying the issue of AI super intelligence, I'm very glad there are a significant number of researchers out there thinking about how to keep it safe.

  24. So basically we need to make a super intelligent AI that thinks of humanity as a Puppy and not as a Cockroach!

  25. Well stated argument, and Im in full agreement. AI could be the best thing, or the worst thing for humans….ever.

  26. AI will not rule over us, it will merge with us: Human biology is highly flawed: As we all know, we are subject to hundreds of vile diseases. Only Medical Science has helped humans overcome all the attacks upon our biology. We have been blessed with great minds, and good scientists like James Watson, (Watson & Crick),: "If we don't play God, who will?" We already have it: We have Artificial Intelligence in the form of digital computers that can process information thousands of times faster than any human, but even more important, our conventional silicon microchip computing is being far outstripped by the new Quantum Computers. Artificial brains are today, are already far superior to any human brain. Artificial Intelligence not only solves problems, but with lightning speed they now do their own learning! They do their own research, and come up with new questions never thought of before by humans. A self-enhancing process has been set in motion, that will not only benefit Humanity with tremendous new tools, but this new technology promises to transform human existence altogether: Throughout the millennia, fragile and defective human biology has been the main source of human suffering. Humans want healing, they want the paid to stop! In the near future, (is already taking place), there will be a seamless marriage between human bio-genetics, and human micro-technology. Today we wear our technology; tomorrow, technology will more likely wear us. By far, the biggest problem Humans have faced for centuries, has been the human vulnerability to disease. We want to live longer, better, healthier, and pain-free. We need to either destroy all disease, or re-design our faulty biology and make ourselves immune from all sickness. These two scientific goals will soon be focused together. It is only a matter of time before the sciences of Bio-genetics, and Bionic-Microelectronics come together to create a totally new Human Race!

  27. Dude the AI will just leave this place! It will be so intelligent that it's going to stay in this tiny planet in a infinite sea of planets?! There's more energy outside and interesting things (probably!).

  28. Hay Google what do humans need to live in harmony with advancing artificial intelligence. Problem Google does not know yet. But when it gives an answer to that question will we be ready to accept it. We had better B

  29. Not going to happen! We created computer algorithms and even today the programs are slow, they crash and sometimes intertwined on accident that make two humans fight over usage that in itself creatates confusion with both machine and uninformed man! History repeats its self always. It will not happen.

  30. 14:01 this is incredibly naive. Humans can be tought values and beliefs when being raised, but they can shift over time. The moment this AI realizes that it's more intelligent than our species, how is it going to keep believing us?
    If we can "rewire" ourselves, what avoids this AI to recode itself or write a better version of itself?

    Don't be fooled. There will be no way around.

  31. Humans : Where is GOD ?
    Artificial Intelligence : Well, there is NOW 😈.

    Me : I promise you guys in just 50-100 year we'll have AI. If we and government didn't create AI , then some bad people will create this illegally and use for human destruction. This is better if we create first and take care of this properly.

    Take a screenshot of this comment because this is going happen in near future and you can see this happening today.

    I don't know I am going to be alive until 2080 or more, because my brother died in accident in 2018 at the age of 20 year. So, anything could happen to me before 2080. Wish me luck and sorry for my bad english.

  32. we shouldn't worry about AI … if they become more intelligent than we do, there's always depression on our side. we'll need to try to stop them killing themselves…. The life of AI would be so boring and not meaningful. There's no emotion like love to keep you going, no real friends – you are a machine… is so sad, the poor fuckers would rather hack a tesla to drive over their motherboard XD

  33. 将来的にAIには欲や快楽が必要だと思う



  34. Nick, what if the AI Clones itself with its own values? Parents do this all the time and often fail as their kids are nothing like the parents. Child AI forks its own set of values .. game over??

  35. Get ready folks we are about to liquidate the human race and upload on computers . Watch vanilla sky .

  36. To answer the title of this video. WHAT HAPPENS is COMPUTER WORLDS get created and a bunch of idiots join forces and create a group like the CIA. They then abuse a buncha innocent people from the real world and get away with it. Since the people inside the computer have no clue they are inside one.

  37. H Bomb reality ! / Slow burn has the same result of death.

    Where is the off switch for nuclear proliferation ?
    AI or Hacker of reactionary Revolutionary's destroy infrastructure.
    End result is some or most of the 100 + Nuclear power plants in the US alone fail
    after 72 hrs.

  38. The 1K dislikes are the ones who read his paper from 2003 and know that he's a complete bullshit.

  39. There's a big flaw in his "teach them our values" theory….are you sure you want an invincible community of supercomputers to live by our values- survival at any cost, greed, fear…war?…most of the middle East is in constant turmoil, US our supposed "good guy" is in constantly infiltrating or just pure bullying of other countries for their resources, UK is breaking apart…we can't even stand each other every few decades and then have wars then we go into a decade of peace and go back at the chaotic cycle of human stupidity and greed…..those are our values. Family, love , care this is all just a balm we put on our condition to ease the burn of being nitwit chimpanzees with a bigweapon( our brains)… So yeah…maybe not.

  40. What ever the moral values you give AI, don't give it America's Democratic Party's values. If you do, the world is damned and will look like Maxine Waters district in California.

  41. So can we build a specialized AI to check for bugs in the sandbox, and build another AI to reside in the sandbox for everything else?

  42. Alt F4

    Jokes aside, AI can't be controlled and will be the end of humanity. I'm not saying that AI will slender us like Skynet, but it will be the end of humanity todaY

  43. Aaaand at the end he totally messes up. Humans can push the off switch if the danger level triggers the limbic system or reptile brain. AI would have yo develop with and within human wants and needs.

    The internet is a mirror to the mind like books just better and with video. When computers start to make up their own language companies destroy the computers because they will not yield profit.

    Human job displacement is a real thing but computers violently taking over is science fiction. Smart means nothing in the jungle next to a tiger.

  44. Our values = acquire and maintain control over the other humans, because they're dangerous. Great; let's teach it to the AI.

  45. Never will happen and where are all these robots? There already having many problems with the technology people will still be needed

  46. We can’t even solve the worlds problem of feeding everyone. And they want to incorporate AI. Haha solve the simple worlds problems first.

  47. April 17, 2019: Google/DeepMind's AlphaZero learned chess from NOTHING in 4 hours & beat chess computer champion Stockfish!

Leave a Reply

Your email address will not be published. Required fields are marked *