Service orchestration for intelligent automated assistant

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to intelligent automated assistant systems. Disclosed is method of operating an intelligent automated assistant. The method is carried out in an electronic device having a processor and memory which stores instructions for execution by the processor. The processor executes instructions on which a user request is received, wherein the user request includes a speech input received from the user. Based on the representation of user intent, a plurality of relevant task parameters are identified for the task flow, wherein the plurality of relevant task parameters includes at least one task parameter explicitly expressed in speech input and at least one task parameter logically derived from context information associated with the speech input.

EFFECT: high accuracy of presenting a user with relevant information by taking into account task parameters logically derived from context information.

11 cl, 50 dwg, 5 tbl

 

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority of the provisional patent application (USA) serial number 61/295774, entitled "Intelligent Automated Assistant", attorney statement number SIRIP003P, filed January 18, 2010, contained in this document by reference.

[0002] This application further claims priority of patent application for invention (USA) 12/987982, entitled "Intelligent Automated Assistant", attorney statement number P10575US1, filed January 10, 2011, contained in this document by reference.

[0003] This application is further related to patent application (USA) serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006, which is contained in this document by reference.

[0004] This application further relates to provisional patent application (USA) serial number 61/186414, entitled "System and Method for Semantic Auto-Completion", filed on 12 June 2009, contained in this document by reference.

The technical field TO WHICH the INVENTION RELATES

[0005] the Present invention relates to intelligent systems, and more particularly, to classes of applications for intelligent automated assistants.

PRIOR art

[0006] Today's electronic devices have vozmozhnostuvelichivat access to substantial, increasing and diverse number of functions, services and information, both via the Internet and from other sources. The functionality for such devices is expanding rapidly, as many home appliance, smartphones, tablet computers, etc. have the ability to run software applications to perform various tasks and provide different types of information. Often, each application, function, web site, or sign has its own user interface and its own practical paradigms, many of which can be complicated to learn or difficult to understand. In addition, many users may find it hard even to detect to what functionality and/or information available on their electronic devices or on various web sites; thus, these users may be disappointed or confused or simply may be unable to effectively use the resources available to them.

[0007] In particular, inexperienced users or people who have any physical impairments or disabilities, and/or are elderly, fidgety, inattentive, and/or operating a vehicle, may experience difficulty regarding effective interaction with your electronic �the equipment and/or effective use of online services. These users, in particular with a high probability should experience difficulty with a large number of diverse and inconsistent features, applications and web sites that might be available for their use.

[0008] Accordingly, existing systems are often difficult to use and navigate and often users are inconsistent and difficult to interpret interfaces that often hinder the effective use of technology by users.

Summary of the INVENTION

[0009] According to different variants of implementation of the present invention, the intelligent automated assistant is implemented on the electronic device to facilitate user interaction with the device and help the user to effectively apply local and/or remote service. In various embodiments, the intelligent automated assistant is in contact with the user in a single conversational manner using dialogue in natural language and, if necessary, activates an external service, to obtain information or perform various actions.

[0010] According to different variants of implementation of the present invention, the intelligent automated assistant integrates a lot of features provided by �various software components (e.g., to support the recognition of a natural language and dialogue in natural language, multimodal input, personal information management, flow management tasks, orchestration of distributed services, etc.). In addition, in order to offer intelligent interfaces and useful functionality to users, the intelligent automated assistant according to the present invention, in at least some embodiments, the implementation can coordinate these components and services. Conversational interface and the ability to obtain information and to perform any further task are implemented, at least in some embodiments, through the coordination of the various components, such as language components, dialog components, components of task management, information management components and/or many external services.

[0011] According to different variants of implementation of the present invention, a system based on the intelligent automated assistant can be configured to provide various special types of operations, functionalities, and/or signs and/or to combine multiple attributes, operations and applications of electronic devices on which they are installed. In some embodiments, a system based on the INTA�intellectual automated assistant of the present invention may perform any or all of the following: active retrieving input from the user, interpreting the user's intentions, the disambiguation between competing interpretations, requesting and receiving clarifying information as needed, and perform (or initiate) actions based on discern intentions. Actions can be performed, for example, by activating and/or interaction with any applications or services that may be accessible on the electronic device, as well as the services that are available through an electronic network such as the Internet. In various embodiments, such activation of the external services can be performed through the API or through any other suitable mechanism. Thus, the system based on the intelligent automated assistant various embodiments of the present invention can unify, simplify and improve user interaction on many different applications and functions of the electronic device and the services that may be accessible on the Internet. The user thereby can be released from the load by learning what functionality may be available in the device and services through a web connection, how to interact with such services, to get what he wants, and how to interpret the output taken from such cases�b; on the contrary, the assistant according to the present invention can act as an intermediary between the user and such diverse services.

[0012] in Addition, in various embodiments, the helper according to the present invention provides a conversational interface that the user may find more intuitive and less cumbersome than traditional graphical user interfaces. The user can contact in the form of spoken dialogue with the assistant using any number of available mechanisms for input and output, such as, for example, speech, graphical user interfaces (buttons and links), text input and etc. the System can be implemented using any number of different platforms, such as devices API, web technologies, e-mail, etc. or any combination thereof. Requests for additional input can be presented to the user in the context of this conversation. Short-term storage and long-term storage device can be used, so that the user input can be interpreted in the proper context based on past events and data exchange in this session, as well as statistical information and profile information about the user.

[0013] in Addition, � various embodiments, context information, extracted from user interaction with indication, operation or application in the device, can be used to simplify other features, operations, or applications in the device or in other devices. For example, the intelligent automated assistant can use the context of the phone call (for example, a call from a man) to simplify the initiation of the text message (for example, to determine what a text message is sent to the same person without the need for the user to explicitly specify the recipient of the text message). Intelligent automated assistant according to the present invention can thus interpret instructions, for example, "send text message", and "him" is interpreted according to the context information extracted from a current phone call and/or any feature, operation, or application in your device. In various embodiments, the intelligent automated assistant takes into consideration the different types of available context data to determine a contact from your address book to use, and what contact to use, what phone number to use for contact, etc., so the user should not re-wook�manifest this information manually.

[0014] In various embodiments, the assistant may also take into account external events and react accordingly, for example, to initiate action, to initiate communication with the user, to provide alerts and/or modify a previously initiated action with regard to external events. If you want input from the user, the conversational interface can again be used.

[0015] In one embodiment, the implementation of the system is based on sets of interrelated subject areas and tasks and uses the additional functionality supported by external services, which can interact with the system. In various embodiments, these external services include services based on web technologies, as well as the functionality associated with the hardware device itself. For example, in a variant implementation, in which the intelligent automated assistant is implemented on a smartphone, personal digital device, a tablet computer, or other device, the helper can control many operations and functions of the device, for example, to dial a telephone number, send a text message, set reminders, add events to the calendar, etc.

[0016] In various embodiments, the system according to the present� to the invention can be implemented to provide assistance in any of a number of different subject areas. Examples include:

- local service (including location based and time services, such as restaurants, movies, ATMs (ATM), activities and places to meet);

service of personal and social memory (including scheduled tasks, notes, calendar events, shared links, etc.);

- e-Commerce (including online purchases of goods such as books, DVDs, music, etc.);

service on travel (including flights, hotels, attractions, etc.).

[0017] the Specialists in the art should recognize that the above list of fields is just approximate. In addition, the system of the present invention can be implemented in any combination of subject areas.

[0018] In various embodiments, the system based on the intelligent automated assistant, disclosed in this document can be adapted to or designed to include functionality to automate the application of data and services available on the Internet to discover, find, choose, buy, reserve or order products and services. In advanced�tion to automate the process of using these data and services, at least one embodiment of the system based on the intelligent automated assistant, disclosed in this document can also provide simultaneous combined use of several data sources and services. For example, it may combine information about products from multiple review web-sites, check out the prices and availability from several distributors and check their location and time constraints, as well as to help the user to find a personalized solution to your problem. Additionally, at least one variant of implementation of the system based on the intelligent automated assistant, disclosed herein can be configured to include functionality to automate the use of data and services available on the Internet to discover, gather information, choose, reserve, and otherwise learn about the practice (including, but not limited to, films, events, performances, exhibitions, shows and attractions; places to go (including but not limited destinations when traveling, hotels and other places, where to stay, attractions and other places of interest, etc.); places to eat and drink (such as restaurants and bars), the times and months�Oh to meet with other people, as well as any other source of entertainment or social interaction, which you can find on the Internet. Additionally, at least one variant of implementation of the system based on the intelligent automated assistant, disclosed herein can be configured to include functionality to provide applications and services through a dialogue in natural language, which can otherwise be provided through special applications with graphical user interfaces, which includes the search (including location-based search); navigation (map and directions for route); search in a database (for example, finding organizations or people by name (name) or other properties); get data regarding weather conditions and weather forecasts, check the prices of commercial items or status of financial transactions; monitoring traffic or the status of flights; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; exchange of data via email or other platforms messaging; and local or remote devices (e.g., dialing the phone, lighting control and temperature control for home protective �the equipment, play music or video, etc.). Additionally, at least one variant of implementation of the system based on the intelligent automated assistant, disclosed herein can be configured to include functionality to identify, generate and/or provide personalized recommendations for actions, products, services, source of entertainment, time management or any other service recommendations, which benefits from the interactive dialogue in natural language and automated access to data and services.

[0019] In various embodiments, the intelligent automated assistant according to the present invention can control many features and operations of the electronic device. For example, the intelligent automated assistant can invoke services that interact with the functionality and applications on the device through APIs or other means to perform the functions and operations, which otherwise can be initiated using the traditional user interface in the device. These functions and operations may include, for example, setting the alarm, making a call, tracetextbox messages or email messages add calendar events, etc., These functions and operations may be performed as additional functions in the context of spoken dialogue between the user and the assistant. These functions and operations may be specified by the user in the context of this dialogue, or they can be automatically performed based on the context of the dialogue. Specialists in the art should recognize that the assistant can thus be used as a control mechanism for initiating and controlling various operations on the electronic device, and can be used as an alternative to traditional mechanisms, such as buttons or graphical user interfaces.

BRIEF description of the DRAWINGS

[0020] the Accompanying drawings illustrate several embodiments of the invention and together with the description serve to explain the principles of the invention according to variants of implementation. Specialists in the art should recognize that the specific variants of implementation, illustrated in the drawings, are merely exemplary and do not intend to limit the scope of the present invention.

[0021] Fig.1 is a block diagram illustrating an example of one embodiment of a system based on the intelligent automated �of amewrica.

[0022] Fig.2 illustrates an example interaction between the user and the intelligent automated assistant according to at least one variant of implementation.

[0023] Fig.3 is a block diagram illustrating a computing device suitable for implementing at least part of the intelligent automated assistant according to at least one variant of implementation.

[0024] Fig.4 is a block diagram illustrating an architecture for implementing at least part of the intelligent automated assistant in a standalone computing system, according to at least one variant of implementation.

[0025] Fig.5 is a block diagram illustrating an architecture for implementing at least part of the intelligent automated assistant in a distributed computing network, according to at least one variant of implementation.

[0026] Fig.6 is a block diagram showing the architecture of the system, illustrating several different types of customers and modes of operations.

[0027] Fig.7 is a block diagram illustrating the client and server that communicate with each other to implement the present invention according to one variant of implementation.

[0028] Fig.8 is a flowchart illustrating a fragment �active ontology according to one variant of implementation.

[0029] Fig.9 is a block diagram illustrating an example alternative embodiment of a system based on the intelligent automated assistant.

[0030] Fig.10 is a block diagram of the sequence of operations illustrating a method of operation for the component(s) of the active extract input according to one variant of implementation.

[0031] Fig.11 is a block diagram of the sequence of operations illustrating a method for active extraction print input according to one variant of implementation.

[0032] Fig.12 to 21 are screen shots illustrating some parts of some procedures for active extraction print input according to one variant of implementation.

[0033] Fig.22 is a block diagram of the sequence of operations illustrating a method for active extraction for voice input or speech input according to one variant of implementation.

[0034] Fig.23 is a block diagram of the sequence of operations illustrating a method for active retrieval of input for input based on the GUI according to one variant of implementation.

[0035] Fig.24 is a block diagram of the sequence of operations illustrating a method for active extract input at the flow level dialogues according to one variant of implementation.

[0036] Fig.25 is a block diagram sequence�eljnosti operations illustrating a method for active monitoring of relevant events according to one variant of implementation.

[0037] Fig.26 is a block diagram of the sequence of operations illustrating a method for active retrieval of multimodal input according to one variant of implementation.

[0038] Fig.27 is a set of screen shots illustrating an example of various types of functions, operations, actions, and/or other signs that may be provided by the component(s) of domain models and orchestration of services according to one variant of implementation.

[0039] Fig.28 is a block diagram of the sequence of operations of a method illustrating an example of a method for natural language processing according to one variant of implementation.

[0040] Fig.29 is a screen shot illustrating a natural language processing according to one variant of implementation.

[0041] Fig.30 and 31 are screen shots illustrating an example of various types of functions, operations, actions, and/or other signs that may be provided by the component(s) CPU thread dialogues according to one variant of implementation.

[0042] Fig.32 is a block diagram of the sequence of operations illustrating a method of operation for the component(s) CPU thread dialogues according to one of variationsummary.

[0043] Fig.33 is a block diagram of the sequence of operations of a method illustrating a process of automatic call and response according to one variant of implementation.

[0044] Fig.34 is a block diagram of the sequence of operations of a method illustrating an example of a task flow for the problem of selection with constraints according to one variant of implementation.

[0045] Fig.35 and 36 are screenshots illustrating example operations of the task of selection with constraints according to one variant of implementation.

[0046] Fig.37 is a block diagram of the sequence of operations of a method illustrating an example procedure to perform the procedure orchestration of services according to one variant of implementation.

[0047] Fig.38 is a block diagram of the sequence of operations of a method illustrating an example procedure service activation according to one variant of implementation.

[0048] Fig.39 is a block diagram of the sequence of operations of a method illustrating an example procedure of the multi-phase output according to one variant of implementation.

[0049] Fig.40 and 41 are screenshots illustrating examples of output processing according to one variant of implementation.

[0050] Fig.42 is a block diagram of the sequence of operations of a method illustrating an example of processing multimodal output according to one embodiment �of sushestvennee.

[0051] Fig.43A and 43B are screenshots illustrating an example of using the component(s) of short-term personal storage device to maintain a dialog context of changing the location, according to one variant of implementation.

[0052] Fig.44A-44C are screenshots illustrating an example of using the component(s) long-term personal storage device, according to one variant of implementation.

[0053] Fig.45 illustrates an example of an abstract model for the problem of selection with restrictions.

[0054] Fig.46 illustrates an example of the flow of dialogue to help guide the user through the search process.

[0055] Fig.47 is a block diagram of the sequence of operations illustrating a method of choice with constraints according to one variant of implementation.

DETAILED DESCRIPTION of embodiments of

[0056] described in detail the various technologies in relation to several exemplary embodiments, as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth to provide a full understanding of one or more aspects and/or characteristics described or indicated by reference into this document. However, for specialists in the art d�should be obvious one or more aspects and/or characteristics described or indicated by reference into this document, can be implemented without some or all of these specific details. In other instances, well known process steps and/or structures are not described in detail so as not to hinder the understanding of certain aspects and/or characteristics described or indicated by reference into this document.

[0057] One or more different inventions may be described in the present application. Additionally, for one or more of the invention(s), many variants of implementation, described herein, can be described in this patent application and are presented only as an illustration. Described variants of implementation does not have the intention to be limiting in any sense. One or more of the invention(I) can be widely applicable to numerous embodiments, as should be obvious from the disclosure of the essence. These implementation options are described in sufficient detail to enable specialists in the art to implement one or more of the invention(I), and it should be understood that may be used in other variants of implementation, and that structural, logical, software, electrical and other changes may be made without deviation about� amount of one or more of the invention(s). Accordingly, specialists in the art should recognize that one or more of the invention(s) can be implemented with various modifications and changes. Specific characteristics of one or more of the invention(s) can be described in relation to one or more particular embodiments or drawings that are part of this disclosure, and in which is shown, as an illustration, specific embodiments of one or more of the invention(s). However, it should be understood that such features are not limited to use in one or more particular embodiments or drawings in respect of which they are described. The present disclosure entity is neither a literal description of all embodiments of one or more of the invention(s) nor the list of features of one or more of the invention(s) that must be present in all variants of implementation.

[0058] the Headings of sections provided in this patent application and the title of this patent application is provided for convenience only and should not be construed as any limitation of disclosure.

[0059] Devices that communicate with each other, are not required to support continuous communication with each other, unless expressly decree�but otherwise. In addition, devices that communicate with each other may communicate directly or indirectly through one or more intermediaries.

[0060] the Description of an embodiment with several components that communicate with each other, does not imply that all such components are required. On the contrary, describes a variety of optional components to illustrate the wide variety of possible embodiments of one or more of the invention(s).

[0061] Further, although process steps, the method steps, algorithms, etc. can be described in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described in this patent application in itself does not indicate the commitment of the stages in this order. The steps of the described processes may be performed in any practical order. Additionally, some steps may be performed simultaneously despite the description or imply as being performed simultaneously (e.g., because one step is described after the other step). In addition, the illustration of the process through its demonstration in the drawing does not imply that proillyustrirovany� process excludes other changes and modifications therein, does not imply that the illustrated process or any of its steps are required for one or more of the invention(s), and does not imply that the illustrated process is preferred.

[0062] When describes a single device or article, it should be apparent that multiple devices/products (interacting or not) can be used instead of a single device/product. Similarly, if describes several devices or products (they interact or not), it should be apparent that a single device/article may be used instead of multiple devices or products.

[0063] the Functionality and/or features of the device may be alternatively implemented by one or more other devices that are not are explicitly described as having such functionality/features. Thus, other embodiments of one or more of the invention(s) need not include the device itself.

[0064] the Techniques and mechanisms described or indicated by reference into this document, for clarity, sometimes described in the singular. However, it should be noted that the specific implementation options include several iterations of the technology or more implementations of the mechanism, unless otherwise indicated.

[0065] Although �opisyvayutsya in the context of technology based on the intelligent automated assistant, can be understood that the various aspects of the technology described in this document (such as, for example, associated with active ontologies), also can be deployed and/or used in other technical fields involving human and/or computer-based interaction with the software.

[0066] Other aspects related to technology based on the intelligent automated assistant (e.g., which may be used by, be provided through and/or be implemented in one or more embodiments of the system based on the intelligent automated assistant, described in this document), are disclosed in one or more of the following contrasted materials:

- provisional patent application (USA) serial number 61/295774, entitled "Intelligent Automated Assistant", attorney statement number SIRIP003P, filed January 18, 2010, the disclosure of which is contained in this document by reference;

- the patent application (USA) serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006, the disclosure of which is contained in this document by reference; and

- provisional patent application (USA) serial number 61/186414, entitled "System and Method for Semantic Auto-Completion", filed on 12 June 2009, revealed� which is contained in this document by reference.

Hardware architecture

[0067] In General, the intelligent automated assistant, disclosed in this document can be implemented in hardware or in a combination of software and hardware. For example, they can be implemented in the operating system kernel, in a separate user process, in a library package bound to network-based applications in a specially designed machine, or a network interface Board. In a particular embodiment of the technology disclosed herein can be implemented in software, for example, in the operating system or application running in the operating system.

[0068] Software/hardware hybrid implementation(I) at least part of option(s) implementation of intelligent automated assistant, disclosed herein can be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in a storage device. These network devices may have multiple network interfaces that can be configured to use various types of data transmission protocols over the network. General architecture for some of these machines may be obvious from the op�attractive sledge, disclosed in this document. According to specific variants of implementation, at least some of the features and/or functionality of various embodiments of intelligent automated assistant, disclosed in this document can be implemented in one or more network host General purpose machines, such as computer system end user, computer, network server or server system, mobile computing device (e.g., personal digital device, a mobile phone, a smartphone, a laptop computer, tablet computer, etc.), consumer electronic device, music player or any other suitable electronic device, router, switch, etc. or any combination thereof. In at least some embodiments, at least some of the features and/or functionality of various embodiments of intelligent automated assistant, disclosed in this document can be implemented in one or more virtualized computing environments (e.g., network computing clouds, etc.).

[0069] Referring now to Fig.3, shows a block diagram illustrating a computing device 60 suitable for implementing at least part of the PR�signs and/or functionalities of the intelligent automated assistant, disclosed in this document. Computing device 60 may be, for example, a computer system end user, network server or server system, mobile computing device (e.g., personal digital device, a mobile telephone, smartphone, laptop computer, tablet computer, etc.), consumer electronic device, music player or any other suitable electronic device or any combination or part. Computing device 60 may be configured to exchange data with other computing devices, such as clients and/or servers via a communication network, for example, via the Internet, using known protocols for this connection (wireless or wired).

[0070] In one embodiment of the implementation, computing device 60 includes a Central processing unit (CPU) 62, interfaces 68, and a bus 67 (e.g., a bus using the standard interaction of peripheral components (PCI)). When operating under control of appropriate software or firmware, the CPU 62 may be responsible for implementing specific functions associated with the functions of a specially configured computing device or machine. For example, in at least one embodiment is�the implementation, personal digital device (PDA) can be configured to act as a system based on the intelligent automated assistant that uses a CPU 62, a storage device 61, 65, and interface(s) 68. In at least one embodiment of implementation, the CPU 62 may be instructed to perform one or more of various types of functions and/or operations of an intelligent automated assistant, under the control of software modules/components which, for example, may include an operating system and any appropriate application software, drivers, etc.

[0071] the CPU 62 may include one or more processors 63 such as, for example, a processor from the family of microprocessors or Motorola or Intel family of microprocessors MIPS. In some embodiments, the processor(s) 63 may include specially designed hardware (for example, a specialized integrated circuit (ASIC), electrically erasable programmable permanent memory (EEPROM), field programmable gate array (FPGA), etc.) to control the operation of computing device 60. In a specific embodiment of the storage device 61 (e.g., non-volatile random access memory (RAM) and/or p�permanent storage device (ROM)) is also part of CPU 62. However, there are many different ways in which the storage device can connect to the system. Block 61 of the storage device can be used for many purposes, such as, for example, caching and/or storing data, programming instructions, etc.

[0072] When used in this document, the term "processor" is not limited to just those integrated circuits referred to in the art as a processor, but in a broad sense means a microcontroller, a microcomputer, a controller with programmable logic, a specialized integrated circuit, and any other programmable circuit.

[0073] In one embodiment of the implementation, interfaces 68 are provided as interface cards (sometimes referred to as "line cards"). In General, they control the sending and receiving of data packets on the computer network and sometimes support other peripherals used with the computing device 60. From interfaces that can be granted, provided the Ethernet interfaces, the interface Protocol frame relay, cable interfaces, DSL interfaces, interfaces Protocol token ring, etc., in Addition, may be granted various types of interfaces such as, for example, a universal serial bus (USB), serial�flax, Ethernet, FireWire, PCI, parallel, radio frequency (RF), Bluetooth™, communication in the area of the near field (for example, with the use of magnetism in the near-field zone), 802.11 (wifi), frame relay, TCP/IP, ISDN, Fast Ethernet interfaces, Gigabit Ethernet interfaces based on the asynchronous transfer mode (ATM), high-speed serial interfaces (HSSI) interfaces POS (POS), interfaces, distributed data transmission through fiber-optic (FDDI), etc. In General, such interfaces 68 may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some cases, volatile and/or nonvolatile memory (e.g., RAM).

[0074] Although the system shown in Fig.3 illustrates one specific architecture for a computing device 60 to implement the technology of the invention described in this document, it is by no means the only architecture devices, which can be realized at least part of the features and technologies described in this document. For example, can be used by an architecture having one or more processors 63 such processors 63 may be present in a single device or distributed across any number of devices. In one�the approaches of implementation, one processor 63 handles the exchange of data and calculations in the routing. In various embodiments, different types of features and/or functionalities of the intelligent automated assistant can be implemented in the system based on the intelligent automated assistant, which includes a client device (e.g., personal digital device or smartphone that is running client software) and server system(s) (for example, server system, described in more detail below).

[0075] Regardless of the configuration of the network device system according to the present invention may use one or more storage devices or memory devices (such as, for example, block 65 of the storage device, executed with the ability to store data, program instructions for the General-purpose network and/or other information associated with the functionality of the intelligent automated assistant, described in this document. Software instructions may control, for example, the operating system and/or one or more applications. Storage device or the storage device can also be configured to save data structures, information taxonomique words information alerts, information about the user's clicks and taps and/or other specific non-program information described herein.

[0076] Because the data and software instructions can be used to implement the systems/methods described herein, at least some embodiments of the network device may include a non-volatile machine-readable data storage media, which, for example, may be arranged to store program instructions, state information, etc. for performing various operations described herein. Examples of such non-volatile computer readable data storage media include, but not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media, such as Opticheskie disks, and hardware devices that are specially designed to store and perform program instructions, such as persistent storage device (ROM), flash memory, storage device on the memristor, random access memory (RAM), etc. Examples of program instructions include both machine code, for example, is formed through compe�of atora, and files containing code top-level, which can be executed by the computer using the interpreter.

[0077] In one embodiment, the implementation, the system of the present invention is implemented in a standalone computing system. Referring now to Fig.4 shows a block diagram illustrating an architecture for implementing at least part of the intelligent automated assistant in a standalone computing system, according to at least one variant of implementation. Computing device 60 includes a processor(s) 63, which is performed by the software for the implementation of intelligent automated assistant 1002. Device 1206 input can be of any type suitable for receiving user input, including, for example, a keyboard, touch screen, microphone (for example, for voice input), mouse, touchpad, trackball, switch with five buttons, a joystick and/or any combination thereof. The unit 1207 of the output may be a screen, a speaker, a printer, and/or any combination thereof. Storage device 1210 may be a random access memory device having a structure and architecture known in the art, for use by processor(s) 63 in the course in�execution of the software. Device 1208 data storage may be any magnetic, optical and/or electrical storage device for storing data in digital form; examples include flash memory, magnetic hard drive, CD-ROM, etc.

[0078] In another embodiment, the implementation, the system of the present invention is implemented in a distributed computing network, for example, in a network having any number of clients and/or servers. Referring now to Fig.5, shows a block diagram illustrating an architecture for implementing at least part of the intelligent automated assistant in a distributed computing network, according to at least one variant of implementation.

[0079] In the layout shown in Fig.5, provided with any number of clients 1304; each client 1304 may execute software to implement the client part of the present invention. In addition, any number of servers 1340 may be provided to process requests received from clients 1304. Clients 1304 and server 1340 can communicate with each other via an electronic network 1361, for example, the Internet. Network 1361 may be implemented using any known network protocols, including, for example, wired and/or wireless protocols.

[0080] in Addition, in one embodiment, implementation, CoE�faith 1340 can call external services 1360 if necessary, to get more information or to save information about the previous interactions with specific users. Communication with external services 1360 may be performed, for example, through a network of 1361. In various embodiments, external services 1360 include services based on web technologies and/or the functionality associated or installed in the hardware device. For example, in a variant implementation, in which the assistant 1002 is implemented on a smartphone or other electronic device, assistant 1002 may obtain information stored in the application calendar (the"Supplement"), in contacts and/or other sources.

[0081] In various embodiments, the assistant 1002 can control many features and operations of the electronic device on which it is installed. For example, assistant 1002 may call external services 1360, which interact with the functionality and applications in your device through the API or through some other means, to perform functions and operations, which otherwise can be initiated using the traditional user interface in the device. These functions and operations may include, for example, setting the alarm, making a call, sending a text message or post�new messages add calendar events, etc., These functions and operations may be performed as additional functions in the context of spoken dialogue between the user and the assistant 1002. These functions and operations may be specified by the user in the context of this dialogue, or they can be automatically performed based on the context of the dialogue. Specialists in the art should recognize that the assistant 1002 can thus be used as a control mechanism for initiating and controlling various operations on the electronic device, which can be used as an alternative to traditional mechanisms, such as buttons or graphical user interfaces.

[0082] for Example, the user can provide input to assistant 1002, for example, "I need to Wake up tomorrow at 8:00". After the assistant 1002 determines the user's intent using the technology described in this document, the assistant 1002 may cause external service 1340 to interact with the function or the alarm clock app in your device. Assistant 1002 sets the alarm on behalf of the user. Thus, the user can use the assistant 1002 as a replacement of traditional mechanisms to set the alarm, or perform other functions in device�TVE. If the user requests are ambiguous or require additional clarification, the assistant 1002 may use various technologies described in this document, including actively extracting, paraphrasing, recommendations, etc., to obtain the necessary information, so called correct service 1340, and the intended action is taken. In one embodiment, the implementation, the assistant 1002 may request in the form of prompting the user for confirmation before calling service 1340 to perform the function. In one embodiment, the implementation, the user can selectively disable the assistant 1002 to call a specific service 1340 or can deactivate all call service if necessary.

[0083] the System of the present invention can be implemented for many different types of clients 1304 and modes of operation. Fig.6 is a block diagram showing the architecture of the system, illustrating several different types of customers and modes of operation. Specialists in the art should recognize that various types of clients 1304 and modes of operation shown in Fig.6, are merely exemplary, and that the system of the present invention may be implemented using clients 1304 and/or modes of operation different from that illustrated. Additionally, �system may include any or all of such clients 1304 and/or modes of operation, one by one or in any combination. Illustrated examples include:

- Computer devices device I / o and/or sensors 1402. The client component can be deployed on any computing device 1402. At least one variant of implementation can be implemented using a web browser 1304A or other application to enable communication with the servers 1340 via the network 1361. The channel input and output can be of any type, including, for example, visual and/or acoustic channels. For example, in one embodiment, the implementation, the system of the invention may be implemented using ways voice communication, providing an implementation option assistant for the blind, which is the equivalent of a web browser is controlled by the speech and uses it to output.

- Mobile device with modules I / o and sensors 1406, for which the client may be implemented as an application on the mobile device 1304B. They include but not limited to mobile phones, smartphones, personal digital devices, tablet devices, networked gaming consoles, etc.

- Appliances with modules I / o and sensors 1410, for which the client may be implemented as a built-in app in the device 1304C.

- Cars and druggernaut tools with interfaces and sensors 1414 dashboard for which the client may be implemented as an embedded system application 1304D. They include but not limited to the car navigation system, voice control system, built-in car electronic multimedia systems, etc.

- Network computing devices, for example, routers 1418 or any other device that is permanently placed or communicates with a network to which the client may be implemented as a permanently locating in device app 1304E.

- Email clients 1424, which implementation option helper connects through the server 1426 modality via e-mail. The server 1426 modality by e-mail acts as a communication bridge, for example, receiving input from the user as e-mail messages sent to the assistant, and sending the output from the helper to the user as answers.

Clients 1428 instant messaging with which variant of implementation of the assistant connects through the server 1430 modality messaging Protocol. The server 1430 modality messaging Protocol acts as a communication bridge that takes input from the user as messages sent to the assistant, and sending the output from a helper to a user as a message in response.

p> - Voice phones 1432, which implementation option helper connects through the server 1430 modality Protocol "speech-in-IP" (VoIP). The server 1430 modality VoIP acts as a communication bridge that takes input from a user as voice, spoken to the assistant, and sending the output from the helper to the user, e.g., as synthesized speech in response.

[0084] For platforms messaging, including but not limited to email, instant messaging, discussion forums, group chat sessions, interactive sessions help or technical support, etc., assistant 1002 may act as a participant in the conversations. Assistant 1002 may monitor the conversation and answer to the people or group with one or more of the technologies and methods described in this document for interactions one-to-one.

[0085] In various embodiments, the functionality to implement the technology of the present invention may be distributed across any number of client and/or server components. For example, software modules may be implemented to perform various functions in connection with the present invention, and such modules can be implemented differently to run on the server and/or client components. Referring now� in Fig.7, shows an example of the client 1304 and server 1340 that communicate with each other to implement the present invention according to one variant of implementation. Fig.7 illustrates one possible arrangement by which software modules can be distributed between the client 1304 and server 1340. Specialists in the art should recognize that the illustrated arrangement is merely exemplary, and that such modules can be distributed in many different ways. In addition, any number of clients 1304 and/or servers 1340 may be provided, and the modules can be distributed between the clients 1304 and/or servers 1340 by any number of different ways.

[0086] In the example of Fig.7, the functionality of input and retrieve functionality output processing is distributed between the client 1304 and server 1340, while the client part of the extraction 1094a input and client processing 1092a output are located in the client 1304 and server part extraction 1094b input and backend processing 1092b output are located on the server 1340. The following components are on the server 1340:

- complete dictionary 1058b;

- comprehensive library of modules 1060b recognition language constructs;

- (master) version 1052b short-term personal storage device;

- main ve�this 1054b long-term personal storage device.

[0087] In one embodiment, the implementation, the client 1304 locally stores a subset and/or parts of these components so as to increase the response speed and reduce the dependence on network communications. Such subsets and/or parts can be maintained and updated according to the known technologies of management of the cache memory. Such subsets and/or parts include, for example:

a subset of the dictionary 1058a;

- a subset of the library modules 1060-a recognition of language constructs;

cache 1052a short-term personal storage device;

cache 1054a long-term personal storage device.

[0088] Additional components can be implemented as part of server 1340, which may include, for example:

- language interpreter 1070;

processor 1080 stream dialogues;

processor 1090 output;

- base 1072 data domain objects;

model 1086 flow problems;

- 1082 orchestration services;

model 1088 characteristics of services.

[0089] Each of these components is described below. The server 1340 receives additional information through interaction with external services 1360 if necessary.

Conceptual architecture

[0090] Referring now to Fig.1, shows a simplified block diagram of a specific exemplary embodiment of intellectual avtomatizirov�tion of assistant 1002. As more fully described herein, various embodiments of the systems based on the intelligent automated assistant can be configured to provide various special types of operations, functionalities, and/or signs, in General, associated with the technology based on the intelligent automated assistant. Additionally, as described in more detail in this document, many different operations, functionality, and/or the characteristics of the system based on the intelligent automated assistant, disclosed in this document can be provided or provide different types of advantages and/or benefits of various objects interacting with the system(s) based on the intelligent automated assistant. An implementation option, shown in Fig.1 may be implemented using any hardware architectures described above, or using another type of hardware architecture.

[0091] for Example, according to different variants of implementation of at least some system(s) based on the intelligent automated assistant can be configured to provide various special types of operations, functionalities and/or features, such as, for example, one or more of the following (or Combe�nation above):

- Automation of application data and services available on the Internet to discover, find, choose, buy, reserve or order products and services. In addition to automating the process of using these data and services, intelligent automated assistant 1002 may also provide simultaneous combined use of several data sources and services. For example, it may combine information about products from multiple review web-sites, check out the prices and availability from several distributors and check their location and time constraints, as well as to help the user to find a personalized solution to your problem.

- Automate the use of data and services available on the Internet to discover, gather information, choose, reserve, and otherwise learn about the practice (including, but not limited to, films, events, performances, exhibitions, shows and attractions; places to go (including but not limited destinations when traveling, hotels and other places, where to stay, attractions and other places of interest, etc.); places to eat and drink (such as restaurants and bars), times and places for meeting with other people, as well as any other sources of entertainment and�and social interaction, which you can find on the Internet.

- Support the work of applications and services through a dialogue in natural language, which otherwise are available through special applications with graphical user interfaces, which includes the search (including location-based search); navigation (map and directions for route); search in a database (for example, finding organizations or people by name (name) or other properties); getting data on weather conditions and weather forecasts, check the prices of commercial items or status of financial transactions; monitoring traffic or the status of flights; accessing and updating calendars and schedules; managing reminders, alerts, tasks and projects; exchange of data via email or other platforms messaging; and local or remote devices (e.g., dialing the phone, lighting control and temperature control for home safety devices, playing music or video, etc.). In one embodiment, the implementation, the assistant 1002 may be used to initiate, operate and control many of the functions and applications available on the device.

- Offer personal recommendations for actions, products, services�, source of entertainment, time management or any other service recommendations, which benefits from the interactive dialogue in natural language and automated access to data and services.

[0092] According to different variants of implementation of at least part of the various types of functions, operations, actions, and/or other features provided by intelligent automated assistant 1002 may be implemented in one or more client systems, in one or more server systems and/or combinations thereof.

[0093] According to different variants of implementation of at least part of the various types of functions, operations, actions, and/or other features provided by assistant 1002 can be realized at least by means of one embodiment of a procedure for the automated call and response, for example, illustrated and described, for example, relative to Fig.33.

[0094] Additionally, various embodiments of assistant 1002, described herein, may include or provide a number of different advantages and/or benefits compared with current technology based on the intelligent automated assistant, such as, for example, one Il� more of the following (or combinations thereof):

- Technology integration understanding natural language and convert the speech into text, which is limited by a set of explicit domain models, tasks, services, and conversations. Unlike technology-based assistant that tries to implement a system of artificial intelligence of General purpose, variants of implementation, described herein, can apply multiple sources of constraints to reduce the number of solutions to more treatable size. This results in fewer ambiguous interpretations of the language, a smaller number of relevant subject areas or tasks, and fewer ways to operationalize the intention in the services. Focusing on specific subject areas, tasks and dialogues also allows to achieve coverage of the subject areas and tasks managed by man dictionary and transformation of intentions in services settings.

- Ability to solve the problems of the user by activating the services on their behalf on the Internet, using an API. Unlike search engines, which return only links and content, some embodiments of the automated assistants 1002, described herein, can automate actions on research and problem solving. Opportunity act�to wirawati several services for this request also provides more functionality to the user what is achieved by visiting a single web site, for example, to create a product or service or find something to do.

- The use of personal information and personal background interaction when interpreting and executing user queries. Unlike traditional search engines or services, answer questions, variants of the implementation described in this document using information from the background personal interaction (for example, background conversations, previous election results, etc.), personal physical context (e.g. user location and time) and the personal information collected in the context of interaction (e.g. name, email address, physical address, telephone number, personal account number, preferences, etc.). The use of these sources of information provides, for example:

- the best interpretation of user input (for example, using personal background and physical context when interpreting language);

- more personalized results (for example, focused on the preferences or the recent elections);

- increased efficiency for the user (e.g., through automation of the stages comprises a check-in services, or filling out forms).

- The use of background conversations in the interpretation of natural language user input. Because the options for implementation may store the personal background and apply understanding of natural language user input, they can also use the dialog context such as current location, time, subject area, stage tasks and task parameters to interpret new inputs. Traditional search engines and shells interpret at least one request regardless of background conversations. The ability to use the background conversations can provide a more natural interaction, which resembles a normal human conversation.

- Active extract of the entry in which the assistant 1002 actively directs and limits the input from the user on the basis of identical models and information used to interpret their input. For example, assistant 1002 may use the model dialogue to recommend the following multi-step dialog with a user, in which they clarify the request; offer to fill partially printed input based depending on the subject area and context opportunities; or to use semantic interpretation to choose from the ambiguous interpr�Tatsiy speech as text or text as intentions.

- Explicit dynamic modeling and manage services in dynamic and reliable orchestration of services. The architecture of the described embodiments provides the ability to assistant 1002 to interact with many external services, to dynamically determine what the service can provide information for a particular user query, to convert the parameters of the user request to different services API, to call several services at once, to integrate the results from multiple services correctly to recover from errors during the operation of services and/or effectively support the implementation of services as their APIs and features are improved.

- Active use of ontologies as a method and hardware system to compose assistants 1002 that simplify software development and maintenance of data systems for the automated attendant. Active ontologies are integration environments, data modeling and implementation for assistants. They provide the infrastructure to link the various sources of models and data (concepts of the subject area, task flows, the dictionary, the recognition engines are language constructs, dialog context, the user's personal information and conversion requests by subject area, isadam in the foreign service). Active ontologies and other architectural innovations described in this paper provide the practicality of constructing deep functionality in subject areas, unification of multiple information sources and services, and accomplish this through a set of subject areas.

[0095] in at least one embodiment, the implementation of intelligent automated assistant 1002 may be configured to use and/or form various special types of data and/or other types of information when performing specific tasks and/or operations. They may include, for example, the input data/information and/or output data/information. For example, in at least one embodiment, the implementation of intelligent automated assistant 1002 may be configured to access, process and/or otherwise utilize information from one or more different types of sources, such as, for example, one or more local and/or remote storage devices, devices and/or systems. Additionally, in at least one embodiment, the implementation of intelligent automated assistant 1002 may be arranged to form one or more different types of output data/information, which, for example, can be stored in for�miniumum device one or more local and/or remote devices and/or systems.

[0096] Examples of different types of input data/information, which may be accessed and/or which can be used by intelligent automated assistant 1002 may include, but not limited to, one or more of the following (or a combination of the above):

- Voice input: from mobile devices such as mobile phones and tablet computers, computers with microphones, Bluetooth headsets, in-car voice system, the phone system, the entries in the service record and send messages to subscribers, voice audiopost services integrated messaging appliances with voice input, such as clock radios, subscriber telephone, a control system home electronic multimedia equipment and video game consoles.

- Enter text with the keyboard on computers or mobile devices, pads on the remote control or other consumer electronic devices, email messages sent to the assistant, instant messaging or similar short messages sent to the assistant, the text taken from players in multiplayer gaming environments, and the text that is sent in the message flows.

- Location information coming from sensors or based on the location�ogenyi systems. Examples include the global location system (GPS) technology using GPS (A-GPS) on mobile phones. In one embodiment, the implementation, the location information is combined with explicit user input. In one embodiment, the implementation, the system of the present invention has the ability to detect when the user is at home, on the basis of known information, the address and the current location. Thus, certain inferences can be made about the type of information that may be of interest to the user, when it is at home in contrast to outside the house, and also about the type of services and activities that should be activated on behalf of the user in whichever he is at home or not.

- Time information from the clock in the client device. It may include, for example, the time from phones or other client devices indicating the local time and time zone. In addition, time can be used in the context of user queries, for example, to interpret phrases such as "in an hour" and "tonight."

- Data compass, accelerometer, gyroscope and/or speed and other sensor data from mobile or handheld devices or embedded systems, for example, a bus�mobile management systems. They can also include data determining the position of devices from remote controls to devices and game consoles.

- Click (click) and select the menu item and other events from a graphical user interface (GUI) on any device that has a GUI. Additional examples include a touchscreen.

- Events from sensors and other data-driven triggers, such as alarms, calendar alerts, triggers price changes, triggers, location, push notifications to your device from servers, etc.

[0097] the Input in the variants of implementation, described herein, also includes the context of the prehistory of user interaction including background conversations and requests.

[0098] Examples of different types of input data/information, which may be accessed and/or which can be used by intelligent automated assistant 1002 may include, but not limited to, one or more of the following (or a combination of the above):

- Text output sent directly to an output device and/or user interface device.

- Text and graphics sent to the user via e-mail.

- Text and graphics sent to the user with �via the messaging service.

- Speech output, may include one or more of the following (or a combination of the above):

-- Synthesized speech.

-- Sampled speech.

-- Recorded message.

- Graphical layout of information with photos, a large amount of text, videos, sounds, and hyperlinks. For example, content, prepared in a web browser.

- The output of an actuator to control the physical actions in the device, such as instructing him to either turn on or off, to make a sound, change color, vibrate, to control lighting, etc.

- The activation of the other apps in your device, for example, the call mapping applications, voice dialing phone numbers, sending e-mail or instant messaging, playing media, adding records to calendars, tasks and applications notes, as well as in other applications.

- The output of an actuator to control the physical actions for devices attached to or controlled by the device such as working with remote camera, control a wheelchair, playing music on remote speakers, video playback on remote displays, etc.

[0099] it is Possible to take into account that intelligent automated assistant 1002 of Fig.1 is only one example of W�wide range of options for implementing the system based on the intelligent automated assistant, which can be implemented. Other embodiments of the system based on the intelligent automated assistant (not shown) may include additional, fewer and/or other components/signs compared to those that are illustrated, for example, in an exemplary embodiment, the system based on the intelligent automated assistant of Fig.1.

User interaction

[0100] Referring now to Fig.2, shows an example interaction between a user and at least one variant of implementation of intelligent automated assistant 1002. The example of Fig.2 assumes that the user is talking with intelligent automated assistant 1002 using the device 1206 input, which may be a mechanism of speech input, and the output is a graphical layout in the device 1207 output, which can be the screen scrollable. The screen 101A of the conversation shows the conversational user interface, demonstrating that said user 101B ("I'd like a romantic place for Italian food near my office), and the response of the assistant 1002, which is a summary of his conclusions 101C ("Well, I found Italian restaurants, which, judging by the reviews, are romantic, close to Your work:") set RES�objectives 101D (shown in the first three from the list of restaurants). In this example, the user clicks the first result in the list, and the result is automatically opened to reveal additional information about the restaurant, shown on the information screen 101E. Information screen 101E and the screen 101A of the conversation can be displayed on one output device, such as a touch screen or other display device; the examples illustrated in Fig.2, are two different States of an output identical to the output device.

[0101] In one embodiment, the implementation, the information screen 101E shows the information gathered and combined from many services, including, for example, any or all of the following:

address and location of organizations;

- the distance from the current location of the user;

reviews from multiple sources.

[0102] In one embodiment, the implementation, the information screen 101E also includes some examples of the services that assistant 1002 can offer on behalf of the user, including:

- dial the phone to call the organization ("call");

- remembering this restaurant for later use ("save");

- sending e-mail messages to anyone with instructions on how to get to, and information about this restaurant ("share");

- showing IU�of topologize and directions how to get to this restaurant on the map ("map");

- save personal notes about this restaurant ("my notes").

[0103] As shown in the example of Fig.2, in one embodiment, the implementation, the assistant 1002 includes intelligent tools, in addition to simple database management applications, such as, for example:

- processing language of intention in natural language 101B, and not just keywords;

- logical conclusion semantic intent of this language input, for example, the interpretation of "Italian food" as "Italian restaurants";

- operationalization of semantic intention into the strategy of using online services and for the implementation of this strategy on behalf of the user (for example, the operationalization of desire romantic places in the strategy check online review sites for reviews that describe the place as "romantic").

Components of intelligent automated assistant

[0104] According to different variants of implementation, the intelligent automated assistant 1002 may include many different types of components, devices, modules, processes, systems, etc., which, for example, can be implemented and/or refined through the use of hardware and/or combination of hardware � software. For example, as illustrated in an exemplary variant of the implementation according to Fig.1, assistant 1002 may include one or more of the following types of systems, components, devices, processes, etc. (or combinations of the above):

- one or more active ontologies 1050;

component(s) 1094 active extract input device may include a client portion 1094a and backend 1094b);

component(s) 1052 short-term personal storage device (may include home version 1052b and cache 1052a);

component(s) 1054 long-term personal storage device (may include home version 1052b and cache 1052a);

component(s) 1056 domain models;

component(s) 1058 dictionary (may include full dictionary 1058b and a subset 1058a);

component(s) 1060 module(s) recognition of language structures (may include full library 1060b and a subset 1560a);

component(s) 1070 language interpreter;

database(s) 1072 data domain objects;

component(s) 1080 processor thread dialogues;

component(s) 1082 orchestration services;

component(s) 1084;

component(s) 1086 models of the task flow;

component(s) 1087 models of the flow of the dialogues;

component(s) 1088 models services;

component(s) 1090 CPU output.

[0105] As described in the hol�and Fig.7, in certain client-server embodiments, some or all of these components can be distributed between the client 1304 and server 1340.

[0106] for illustrative purposes, hereinafter described in more detail, at least a portion of the different types of features in a specific exemplary embodiment of intelligent automated assistant 1002 in relation to the exemplary embodiment of intelligent automated assistant 1002 of Fig.1.

Active ontology 1050

[0107] the Active ontology 1050 serve as a unifying infrastructure that integrates models, components and/or data from other parts of embodiments of intelligent automated assistants 1002. In the field of engineering Informatics, ontologies provide a structure to represent data and knowledge, such as classes/types, relationships, attributes/properties and their concretization in the instances. Ontologies are used, for example, to compose the model data and knowledge. In some embodiments, the intelligent automated system 1002, ontologies are part of the infrastructure of the simulation, in which you can compose the model, for example, domain models.

[0108] In the context of the present invention, the "active ontology 1050 may also serve as �Reda run in which different processing elements of the ontological way (e.g., in the presence of various attributes and relations with other processing elements). These processing elements perform at least some tasks intelligent automated assistant 1002. May be provided any number of active ontology 1050.

[0109] in at least one embodiment of implementation, the active ontology 1050 can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

• Perform the functions of the simulation environment and development, integrating models and data from different component models and data, including, but not limited to:

model 1056 subject areas

- dictionary 1058

- base 1072 data domain objects

model 1086 flow task

model 1087 flow dialogs

model 1088 characteristics of services

• Acting as the data modeling environment, which can work editing tools based on ontologies, to develop new models, data structures, schema, and database view.

• Perform the functions of the runtime live, to�critisise values for the elements of domain models 1056, tasks 1086 and/or dialogues 1087, recognition modules language structures and/or vocabulary 1058, and for specific user information, for example, are in short-term personal storage device 1052, long-term personal storage device 1054 and/or in the results of orchestration services 1182. For example, some nodes active ontology may correspond to the concepts of the subject area, such as a restaurant and its property in the name of the restaurant". In the course of performing live, these active nodes of the ontology can be specified by using the identification data of the specific object of the restaurant and its name, and its name matches the words in the input utterance in a natural language. Thus, in this variant implementation, the active ontology serves as a simulation environment, indicating the concept that restaurants are objects with identity, which have names, and saving the dynamic relations of these modeling nodes with data from database objects and grammatical analysis of natural language.

• Enable communication and coordination between components and processing elements of the intelligent automated assistant, such as, for example, one or more of the following (or com�Inacio above):

component(s) 1094 active extract input

component(s) 1070 language interpreter

component(s) 1080 processor flow dialogs

component(s) 1082 orchestration services

component(s) 1084 services

[0110] In one embodiment, the implementation of at least some of the functions, operations, actions, and/or other evidence of active ontology 1050, described herein, can be implemented, at least partially, using a variety of methods and hardware systems described in the patent application (USA) serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006.

[0111] in at least one embodiment of the implementation, this instance of active ontology 1050 may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be access through active ontologies 1050 may include, but not limited to, one or more of the following (or a combination of the above):

static data that is available from one or more components �intellectual automated assistant 1002;

- data that is dynamically specified in a per user session, for example, but not only, the maintenance status for specific user inputs and outputs is transferred between components of intelligent automated assistant 1002, the contents of short-term personal storage device, the logical conclusions of the previous States of a user session, etc.

[0112] Thus, the active ontology 1050 are used to unify the elements of the various components of intelligent automated assistant 1002. Active ontology 1050 enables author, designer or developer of a system to integrate components so that the elements of one component are identified with elements of other components. The author, designer or developer of the system is thereby able to combine and integrate components in a simpler manner.

[0113] Referring now to Fig.8, shown is an example of the active ontology 1050 according to one variant of implementation. This example has the intention to help illustrate some of the various types of functions, operations, actions, and/or other signs that may be provided through the active ontology 1050.

[0114] the Active ontology 1050 in Fig.8 includes �you represent restaurant and activities with eating. In this example, the restaurant is a concept 1610 with properties such as its name 1612, offer cuisine 1615 and its location 1613, which, in turn, can be modeled as a structural node with properties for actual address 1614. The concept of the event with food intake can be modeled as a node 1616, which includes dinner 1617 in the restaurant (which has a number 1619 participants) and the period 1618-time.

Active ontologies may include and/or reference model 1056 subject areas. For Example, Fig.8 illustrates the model 1622 subject area dinners in the restaurants associated with the concept of 1610 restaurant and concept 1616 activities with eating. In this case, the active ontology 1050 includes model 1622 subject area dinners in restaurants; in particular, at least two node active ontology 1050, namely, restaurant 1610 and the event 1616 with the meal are also included and/or specified by a reference in the model 1622 subject area dinners in restaurants. This a domain model is, among other things, this idea that a dinner party in the restaurant embodies the event with food intake that occurs in restaurants. The active nodes of the ontology of the restaurant 1610 and the event 1616 with the meal also included and/or specified by reference to other component�x intelligent automated assistant, shown by dotted lines in Fig.8.

Active ontologies may include and/or refer to models 1086 flow problems. For Example, Fig.8 illustrates the model 1630 flow tasks, organizing events, which models the organization of events, regardless of subject areas, used to depend on the subject area type of event: event 1616 with eating. Here, the active ontology 1050 includes a General model 1630 flow tasks, organizing events, which contains nodes representing activities and other concepts involved in their organization. Active ontology 1050 also includes a host of activities 1616 with food intake, which is a specific type of event. In this example, the event 1616 with meals included or is specified by a reference in the model 1622 subject area in the model 1630 flow problems, and both of these models are included and/or specified by the link in the active ontology 1050. On the other hand, the event 1616 with a meal is an example of how active ontology can unify the elements of the various components included and/or referenced by a link in other components of intelligent automated assistant, shown by dotted lines in Fig.8.

Active ontologies may include and/or refer h� model 1087 flow of dialogues. For Example, Fig.8 illustrates the model 1642 flow of dialogues to obtain values of the constraints required for the transactions specified in number of participants soiree in the form of restriction, as represented in the concept 1619. On the other hand, active ontology 1050 provides the infrastructure to link and unify the various components, for example, the model 1087 flow of dialogues. In this case, the model 1642 flow dialogs have a General concept of constraint that is specified in this particular example, as the number of 1619 participants soiree in the form of active node of the ontology. This particular model 1642 flow of dialogue operates at the abstraction of constraints that are independent of the subject area. Active ontology 1050 represents a property 1619 number of participants soiree node 1617 soiree, which is associated with the node 1616 activities with eating. In this embodiment, the implementation of intelligent automated assistant 1002 uses the active ontology 1050, to unify the concept of constraints in the model 1642 flow of dialogue with a property number 1619 participants soiree as part of the cluster of nodes representing the concept 1616 activities with eating, which is part of the model 1622 subject area for dinners in restaurants.

Active ontologies may include�you in and/or referenced in the model 1088 services. For Example, Fig.8 illustrates a service model 1672 restaurant reservation associated with the phase flow of the dialogues to obtain the values required for the service to complete the transaction. In this case, the model 1672 services for reservation services of the restaurant indicates that the reservation requires a value for the number 1619 participants soiree (the number of people sitting at a table, you need to reserve). Number 1619 participants soiree in the form of a concept, which is part of the active ontology 1050, also connects or communicates with the General model 1642 flow of dialogues to a user request regarding restrictions for transactions; in this case, the number of participants soiree is the specified limit for the model 1642 flow dialogs.

Active ontologies may include and/or reference base 1072 data domain objects. For Example, Fig.8 illustrates the database objects in the subject area restaurants 1652 associated with the node 1610 restaurant in the active ontology 1050. Active ontology 1050 is an overall concept of the restaurant 1610, which can be used by various components of intelligent automated assistant 1002, and he further specified through data about specific restaurants in the database 1652 data re�tornow.

Active ontologies may include and/or reference databases 1058 data. For Example, Fig.8 illustrates the vocabulary base 1662 data cuisines, for example, Italian, French, etc., and words associated with each kitchen, for example, "French", "continental", "provincial", etc. Active ontology 1050 includes a node 1610 restaurant, which is associated with the node 1615 offer cuisines that are associated with the presentation of the dishes our database 1662 data cuisines. A specific record in the database 1662 data for the kitchen, for example, "French", thereby connected through the active ontology 1050 as an instance of the concept 1615 offer cuisines.

Active ontologies may include and/or refer to any database that can be transformed into concepts or other concepts in the ontology 1050. Base 1072 data domain objects and lexical databases 1058 data are just two examples of how active ontology 1050 can integrate databases with each other and with other components of the automated assistant 1002. Active ontologies enable the author, designer or system designer to specify a non-trivial transformation between views in the database and the concepts in the ontology 1050. For example, the database schema for database 1652 data restoranuose to represent the restaurant as a table of strings and numbers, or as a projection from a larger database of organizations or any other representation, a suitable base for 1652 data. In this exemplary active ontology 1050, restaurant 1610 is a node of a concept with properties and relationships, well-organized from database tables. In this example, the nodes of the ontology 1050 associated with the elements of the schemas in the database. Integration of database and ontology 1050 provides a unified view for the interpretation and impact on specific data records in the databases from the point of view of large sets of models and data in active ontology 1050. For example, the word "French" can be an entry in the database 1662 data cuisines. Since in this example, the base 1662 integrates data in active ontology 1050, the same word "French" also has an interpretation as possible of the proposed kitchen in the restaurant, which is included in the organization of events with the meal, and the kitchen serves as a constraint that should be used when using the services, booking restaurants etc. Active ontology thus can integrate the database into the simulation environment and perform to interact with the other components of the automated assistant 1002.

[0115] As described above, the active ontology 1050 enables author, designer or system designer to integrate the components; thus, in the example of Fig.8, component elements�yente, such as the limit in the model 1642 flow of the dialogues can be identified with elements of other components, such as the mandatory service parameter 1672 restaurant reservation.

[0116] the Active ontology 1050 may be implemented as, for example, configurations, models, databases, and components in which the relationship between models, databases, and components are any of the following:

- regarding the inclusion and/or implication;

- relationship with links and/or pointers;

interop API as internal regarding the program and between programs.

[0117] for Example, referring now to Fig.9, shows an example alternative embodiment of a system 1002 based on the intelligent automated assistant, which components of the model 1056 subject areas, vocabulary 1058, modules 1060 recognition of language structures, short-term personal storage device 1052 and long-term personal storage device 1054 are arranged within a common container associated with the active ontology 1050, and other components, such as component 1094 active extract input language interpreter 1070 and 1080 processor flow dialogs, associated with the active ontology 1050 via API.

Component(s) 1094 active and�attraction type

[0118] in at least one embodiment, the implementation component(s) 1094 active extract input (which, as described above, can be implemented in a standalone configuration or in a configuration that includes both server and client components) can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

• Extraction, simplification and/or processing input from the user or from the user environment and/or information regarding his need(s) or request(s). For example, if the user seeks to find a restaurant, the extraction module input may obtain information on limitations or user preferences by location, time, cuisine, price, etc.

• Simplification of the various types of input from a variety of sources, such as, for example, one or more of the following (or a combination of the above):

- input from keyboard or any other input device that generates a text

- input from keyboard in user interfaces that offer dynamic recommended fill partial input,

- the input from voice or speech input,

- input from a graphical user in�of erbasov (GUI), in which users click, select, or otherwise directly manipulate graphical objects to indicate the choices,

- input from other applications that form the text and send it to the automated assistant, which includes email, text messaging or other text communication platform.

[0119] By performing the active extract input assistant 1002 has the opportunity to resolve the ambiguity of intent in the early phase of input processing. For example, in a variant implementation, in which the input is provided by speech, signals may be sent to the server 1340, which are extracted words and semantic interpretation. The results of such semantic interpretations can then be used to manage the active extract input that can offer the user alternatives words from which to choose on the basis of their degree of semantic matching, as well as phonetic matching.

[0120] in at least one embodiment, the implementation component(s) 1094 active extract input is active, automatically and dynamically guides the user in the direction of the glands, which can affect one or more of the services offered through options of implementation� " s assistant 1002. Referring now to Fig.10 shows a block diagram of the sequence of operations illustrating a method of operation for components 1094 active extract input according to one variant of implementation.

[0121] the Procedure begins at stage 20. In step 21, the assistant 1002 can offer interfaces on one or more input channels. For example, the user interface may offer the user options to say, or print, or touch at any stage of conversational interaction. In step 22, the user selects the input channel through trigger input in one modality, for example, pressing the button to start the recording or cause the interface to keyboard input.

[0122] in at least one embodiment of implementation, the assistant 1002 offers recommendation default for the selected modality 23. In other words, he suggests 24 that are relevant in the current context, prior to the user any input in this modality. For example, in the modality of text entry, assistant 1002 can offer a list of common words that should start the text queries or commands, such as, for example, one or more of the following (or a combination of the above): imperative verbs (e.g. find, buy, reserve, receive, call, check, will be� schedule, etc.) nouns (e.g., restaurants, movies, events, organizations, etc.) or options in the style menu items with the names of the subject areas of discourse (e.g., weather, sports, news, etc.).

[0123] If the user selects one of the options by default at stage 25, and set a preference to auto-complete 30, the procedure may return immediately. This is a similar operation a traditional choice of menu item.

[0124] However, the initial option may be accepted as partial input, or the user can begin to perform partial input 26. At any point of entry, in at least one embodiment of the implementation, the user can select to indicate that a partial entry is over 22, which leads to the return of the procedure.

[0125] In step 28 the last input that was selected or entered is added to the cumulative input.

[0126] In step 29, the system recommends the following possible inputs that are relevant, given the current input and other sources of restrictions on what is relevant and/or meaningful input.

[0127] in at least one embodiment of implementation, the sources of constraints on user input (for example, which are used in the stages 23 and 29) represent one or more of the different models and data sources that can b�you included in the assistant 1002, that may include, but not limited to, one or more of the following (or a combination of the above):

- Dictionary 1058. For example, it can be recommended word or phrase that matches the current input. In at least one embodiment of implementation, the dictionary can be associated with any or one or more active nodes of ontologies, domain models, task models, models, dialogues and/or model services.

Model 1056 subject areas, which can limit the inputs that may specify or otherwise be consistent with the domain model. For example, in at least one embodiment of implementation, the model 1056 subject areas can be used for recommending concepts, relations, properties and/or instances that are assumed, consistent with the current input.

Modules 1060 recognition of language constructs that can be used to recognize idioms, phrases, grammatical constructions or other structures in the current input, and can be used to recommend filling that fill the structure.

- Base 1072 data domain objects, which can be used to recommend possible objects in the subject area that the exact�Ute with the entry (e.g., names of organizations, titles of movies, names of events, etc.).

- Short-term storage device 1052, which can be used to reconcile any previous input or part of the previous input and/or any other property or fact about the history of user interaction. For example, partial input can be compared with the cities that the user encounters in a session, whether hypothetically (for example, referred to in the requests) and/or physically (e.g., as determined from location sensors).

- In at least one embodiment of the implementation, semantic rephrasing of recent inputs, query, or results can be compared with the current input. For example, if the user has previously executed the query "live music" and got a list of concerts and then printed music in the environment of the active extract input, recommendations may include "live music" and/or "concerts".

- Long-term personal storage device 1054, which can be used to recommend similar items from long-term storage device. Such matching elements may include, for example, one or more or any combination of the following: the domain objects, which are stored (for example, "loved�" restaurants, movies, theatres, venues, etc.), the elements of the list classes, list items, calendar entries, names of people in your contact lists/address books, names of streets or cities mentioned in the list of contacts/address books, etc.

Model 1086 flow task, which can be used in order to recommend to the inputs based on the following possible stages in the task flow.

Model 1087 flow of dialogue, which can be used in order to recommend to the inputs based on the following possible stages in the flow of conversations.

Model 1088 characteristics of services that can be used to recommend possible service for use by title, category, characteristic, or any other property in the model. For example, the user can type part of the name preferred review web site, and the assistant 1002 may recommend the team to fill this request review web site on the subject of the review.

[0128] in at least one embodiment, the implementation component(s) 1094 active extract input is the conversational user interface, for example, an interface in which the user and the assistant communicate by alternately reciting phrases in a conversational manner. Component(s) 1094 active extract input can be performed in�the opportunity to perform and/or implement various types of conversational interfaces.

[0129] in at least one embodiment, the implementation component(s) 1094 active extract input can be performed with the opportunity to perform and/or implement various types of conversational interfaces, in which the assistant 1002 uses the branch of the conversation to prompt as the prompt information from the user according to the models of dialogues. Model dialogues can present a procedure for dialogue, such as, for example, the sequence of the steps required to extract the information required to run the service.

[0130] in at least one embodiment, the implementation component(s) 1094 active extract input limitations and offers guidance to the user in real time at the time when the user is in the process of typing, casting or other input creation. For example, the active extract can direct the user to print the entries of the text that can be recognized through an embodiment of assistant 1002, and/or which may be served through one or more services offered by means of embodiments of assistant 1002. This is an advantage compared to passive waiting for input without restriction from the user as it gives the opportunity to focus user�I on inputs, which can be useful, and/or this enables implementation options of the assistant 1002 to apply their interpretation of the input in real time as the user enters it.

[0131] At least a part of functions, operations, actions, and/or other evidence of active extract input described in this document can be implemented, at least partially, using a variety of methods and hardware systems described in the patent application (USA) serial number 11/518292, entitled "Method and Apparatus for Building the Intelligent Automated Assistant", filed September 8, 2006.

[0132] According to a specific implementation options, multiple instances or threads of the component(s) 1094 active extract input may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software.

[0133] According to different variants of implementation, one or more different threads or instances of the component(s) 1094 active extract input may be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of the component�TA(s) 1094 active extract input. Various examples of conditions or events that may initiate or start the implementation of one or more different threads or instances of the component(s) 1094 active extract input may include, but not limited to, one or more of the following (or a combination of the above):

- The beginning of a user session. For example, when a user session starts, which is a variant of the implementation of assistant 1002, the interface may offer the possibility for the user to initiate an input, e.g., by pressing the button to initiate voice input, or click the text field to initiate a session of typing.

- User input detected.

- When the assistant 1002 clearly requests in the form of prompting the user for input, like when he asks for the answer to a question or offers a menu of the following stages, from which to choose.

- When the assistant 1002 helps the user to perform the transaction and collects data for this transaction, for example, filling out a form.

[0134] in at least one embodiment of the implementation, this instance of a component(s) 1094 active extract input may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation�tion, at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be accessed through the component(s) 1094 active extract input may include, but not limited to, one or more of the following (or a combination of the above):

- database of possible words to use in a text input;

grammar of possible phrases to use when pronouncing text;

- a database of possible interpretations of the speech input;

- a database of previous inputs from the user or from other users;

data from any of the various models and data sources that may be part of embodiments of assistant 1002, which may include, but not limited to, one or more of the following (or a combination of the above):

model 1056 subject areas.

- dictionary 1058;

modules 1060 recognition language constructs;

- base 1072 data domain objects;

- short-term storage device 1052;

- long-term personal storage device 1054;

model 1086 flow problems;

model 1087 stream dialogues;

model 1088 characteristics of services.

[0135] According to different variants of implementation, component 1094 active extract input can apply procedures for active extraction, for example, to one or more of the following (or combinations of the above):

- the printed input;

- speech input;

- input from graphical user interfaces (GUI), which includes gestures;

- the input from the recommendations suggested in the dialogue; and

- events from the computing and/or chuvstvennyh environments.

Active extract printable typing

[0136] Referring now to Fig.11 shows a block diagram of the sequence of operations illustrating a method for active extraction print input according to one variant of implementation.

[0137] Method 110 begins. Assistant 1002 111 takes the partial text entry, for example, through the device 1206 input. Partial text entry may include, for example, the characters printed in the text input field. At any time the user can specify that the printed finished entering 112, for example, by pressing the Enter key. If not finished, the driver generates recommendations 114 versions of the recommendations 116. These recommendations can be syntactic, semantic, and/or other types of recommendations on the basis of any of the sources of information, or limitations described herein. If the recommendation 118 is selected, enter the pre�brushetta 117, it includes the selected recommendation.

[0138] in at least one embodiment of implementation, the recommendations may include extensions to the current input. For example, the recommendation to "rest" can be "restaurants".

[0139] in at least one embodiment of implementation, the recommendations may include replacing parts of the current input. For example, the recommendation to "rest" can be "places to eat".

[0140] in at least one embodiment of implementation, the recommendations may include substitution and rephrasing parts of the current input. For example, if the current entry is "find restaurants in the style of", the recommendation may be "Italian", and when a recommendation is selected, all input can be overwritten as "find Italian restaurants".

[0141] in at least one embodiment of the implementation, the resulting input that is returned is annotated 119, so that the information on what options are selected in step 118, is stored together with the text input. It provides the ability to associate, for example, semantic concepts or objects underlying the strings, with the string it returns, which increases the accuracy of the subsequent language interpretation.

[0142] Referring now to Fig.12-21, shown screen shots illustrating some parts of some procedures for�tive extraction print input according to one variant of implementation. Screen shots illustrate an example embodiment of assistant 1002, implemented on the smartphone, such as iPhone, Apple Inc., Cupertino, California. Input is provided to the device via the touch screen that includes on-screen keyboard functionality. Specialists in the art should recognize that the screenshots illustrate an implementation option, which is just exemplary, and that the technology of the present invention can be implemented in other devices and using other layouts, and layouts.

[0143] In Fig.12, the screen 1201 includes a high-level set of recommendations 1202 shown, when the input is not provided in field 1203. This corresponds to the stage 23 without the input of Fig.10, applied to the step 114 of Fig.11, when there is no input.

[0144] In Fig.13 screen 1301 illustrates an example of using the dictionary to offer recommended fill 1303 partial user input 1305 made in the field 1203 using the on-screen keyboard 1304. These recommended fill 1303 can be a part of the functions 1094 active extract input. The user performs a partial user input 1305 that includes the string "link". Component 1058 dictionary provides convert this string into three different types of instances that p�reciclada as recommended fill 1303: the phrase "public and local activities" is a category of the subject area of activities; summary of commercial information" is a category of the subject area finding local organizations, and "Jewish social club" is the name of the instance of local organizations. Component 1058 dictionary can provide data search and management of namespaces, such as these. The user can touch the button 1306 Go to indicate that he ended input; it instructs the assistant 1002 to continue filled with a text string as the unit of user input.

[0145] In Fig.14 screen 1401 illustrates an example in which the recommended semantic filling 1303 for the partial string "wh" 1305 include the full phrase with the print parameters. These types of recommendations can be ensured through the use of one or more of various models and sources of constraints on the input described in this document. For example, in one embodiment, the implementation shown in Fig.14, "what's going on in the city" is active parameter extraction position for the subject area local activities; "where is the name of the organization is actively removing restrictions in the form of the organization name for the subject area finding local organizations; "as demonstrated in the name of the venue" is active extraction limit as h�the title of the venues for the subject area local activities; and "what's on in the cinema" is active extraction restrictions in the form of the name of the cinema for the subject area local events. These examples illustrate that the recommended fill is formed by means of models, not just retrieved from the database of previously entered requests.

[0146] In Fig.15 screen 1501 illustrates the continuation of the identical example, after the user enters additional text box 1305 1203. Recommended fill 1303 are updated so that they match with additional text 1305. In this example, used data from the 1072 data domain objects: the venue, the title of which begins with "f". It should be noted that this represents a much smaller and more semantically relevant set of recommendations than all the words that start with "f". On the other hand, the recommendations are generated by applying the model, in this case the domain model, which represents the local events taking place in venues which are organizations with names. Recommendations actively check the bushings that need to be potentially significant entries when you use the services of local events.

[0147] Fig.16 screen 1601 illustrates the continuation of the identical p�imera after the user selects one of the recommended padding 1303. Active extract continues through prompts the user to optionally specify the type of information requested, here by presenting a certain number of specifiers 1602, from which the user can choose. In this example, these specifiers are formed through domain models, task flow and the flow of conversations. The subject area are local events that include the categories of events that occur on the specified dates at the specified locations and have names of events and parties performances. In this variant of implementation, the fact that these five options are offered to the user is generated from a model of the flow of the dialogues, which indicates that users should inquire as to the limitations that they have not yet entered, and from the service model, which indicates that these five constraints are parameters for providing services list of local activities available for assistant. Even the preferred choice phrases that should be used as qualifiers, for example, "by category" and "participate", is formed from dictionary databases by subject area.

[0148] Fig.17 screen 1701 illustrates the continuation of the identical example pic�e, the user selects one of the specifiers 1602.

[0149] In Fig.18 screen 1801 illustrates the continuation of the identical example in which the selected qualifier added in 1602 1203, and submitted additional specifiers 1602. The user can select one of the specifiers 1602 and/or to provide additional input text via the keyboard 1304.

[0150] Fig.19 1901 illustrates the continuation of the identical example in which the selected qualifier added in 1602 1203, and submitted additional specifiers 1602. In this example, the previously imposed restrictions not actively extracted redundantly.

[0151] Fig.20 screen 2001 illustrates the continuation of the identical example in which the user touches the button 1306 Go. User input is shown in field 2002, and the message shown in field 2003, providing feedback to the user regarding the query executed in response to user input.

[0152] Fig.21 screen 2101 illustrates the continuation of the identical example in which the results found. The message is displayed in a field 2102. The results of 2103, which includes the input elements that allow the user to view more detailed information, store the identified event, to buy tickets, add notes, etc.

[0153] In one screen 2101 and each�e display screens may be scrolled, allowing the user to scroll upwards to see the screen 2001 or other previously submitted to amend the request if necessary.

Active extraction of speech input

[0154] Referring now to Fig.22 shows a block diagram of the sequence of operations illustrating a method for active extraction for voice input or speech input according to one variant of implementation.

[0155] the Method begins 221. Assistant 1002 receives 121 voice or speech input in the form of a sound signal. Service 122 of converting speech to text or the processor generates a set of options 124 interpretations of the text from the audio signal. In one embodiment, the implementation, the service 122 of converting speech to text is implemented using, for example, recognition module nuances offered by company Nuance Communications, Inc., Burlington, MA.

[0156] In one embodiment of implementation, the assistant 1002 uses statistical language models to generate options 124 interpretations of the text to speech input 121.

[0157] in Addition, in one embodiment, the implementation of the statistical language model is configured to look up words, names and phrases that appear in different models of assistant 1002, shown in Fig.8. For example, in at least one embodiment, the implementation of the statistical language�new models include the words, names and phrases from part or all of the following: the model 1056 subject areas (for example, words and phrases associated with restaurant and events with the meal), model 1086 flow problems (e.g., words and phrases associated with the event organization), model 1087 flow of conversations (for example, words and phrases related to restrictions that are necessary in order to collect inputs for redundancy restaurant), base 1072 data domain objects (for example, names of restaurants), lexical databases 1058 data (e.g., names), model 1088 the services (e.g., names of service providers (such as OpenTable), and/or any words, names or phrases associated with any node of the active ontology 1050.

[0158] In one embodiment of the implementation, statistical language models are also configured to look up words, names and phrases from long-term personal storage device 1054. For example, in statistical language models may be presented with the text of list items, list items, personal notes, calendar entries, names of people in your contact lists/address books, email addresses, street names or cities mentioned in the list of contacts/address books, etc.

[0159] Component ranking analyzes variants of interpretation 124 and ranks them by 126, NASK�only well they correspond to syntactic and/or semantic models of intelligent automated assistant 1002. Can be used any sources of constraints on user input. For example, in one embodiment, the implementation, the assistant 1002 may rank the output of the interpreter to convert your speech into text according to how well executed the grammatical analysis of interpretations in syntactic and/or semantic sense, the domain model, the model of task flow and/or model dialogues, etc.: it evaluates how well different combinations of words in interpretations of text 124 correspond to the concepts, relations, objects and properties of active ontology 1050 and its associated models. For example, if the service 122 of converting speech to text generates two variant interpretations of "Italian food for lunch and Italian footwear to dinner," ranking by semantic relevance 126 may rank "Italian food for lunch" above, if it better matches the nodes in the active ontology 1050, assistant 1002 (e.g., the word "Italian", "food" and "lunch" coincide with nodes in the ontology 1050, and they are all connected by relationships in the ontology 1050, whereas the word "shoes" does not match the ontology 1050 or coincides with a node that is not part of the network subject area dinners in restaurants).

[0160] In various embodiments, algorithms or procedures used by p�the assistant 1002 to interpret the text inputs, includes any variant of the procedure of natural language processing, shown in Fig.28, can be used to rank and quantify options 124 interpretations of the text generated by the service 122 of converting speech to text.

[0161] In one embodiment, the implementation, if the component ranking 126 128 determines that the interpretation of speech with the highest rank of interpretations 124 is ranked above a given threshold, the interpretation with the highest rank may be automatically selected 130. If none of the interpretations is ranked above a given threshold, possible interpretations of the speech 134 are presented to the user 132. The user can then choose 136 between the display options.

[0162] In various embodiments, the user selection 136 between the displayed choices can be obtained by any mode, including, for example, any of the modes of multimodal input described in connection with Fig.16. Such modes include, without limitation, actively recoverable printable enter 2610, actively extracted speech input 2620, actively presents GUI for entering 2640, etc. In one version of implementation, the user may select from options 134 interpret the data a transponder�tions, for example, by touching or reciting. In the case of pronouncing a possible interpretation of the new speech input is greatly limited by a small set of suggested options 134. For example, if you are prompted "do You mean Italian food or Italian shoes?", the user can simply say "food", and the helper can put it together with the phrase "Italian food" and not get confusion with other global interpretations of the input.

[0163] Regardless of the input 130 is selected automatically or 136 is selected by the user, the resulting input 138 is returned. In at least one embodiment of implementation, the returned entry is annotated 138, so that the information on what options are selected in step 136, is saved together with the text input. It provides the ability to associate, for example, semantic concepts or objects underlying the strings, with the string it returns, which increases the accuracy of the subsequent language interpretation. For example, if "Italian food" is proposed as one of the options 134 interpretations on the basis of semantic interpretation Cuisine=Italian food, machine-readable semantic interpretation may be sent together with the user selecting the string "Italian food" as annotated �text 138.

[0164] in at least one embodiment of the implementation, the options 124 interpretations of the text are based on the interpretations of the speech taken as output of the service 122 of converting speech to text.

[0165] in at least one embodiment of the implementation options 124 interpretations of the text are formed by rephrasing interpretation of speech from the point of view of their semantic meaning. In some embodiments, there may be several perefrazirovanie identical interpretation of speech, recommending different meaning of a word or alternatives of the homonym. For example, if the service 122 conversion of speech into text indicates "place to meet", versions of the interpretations presented to the user, can be paraphrased as "a meeting place (a local organization)" and "a meeting place (restaurants)".

[0166] in at least one embodiment of the implementation, the options 124 interpretations of the text include recommendations to correct substring.

[0167] in at least one embodiment of the implementation, the options 124 interpretations of the text include recommendations to correct substring variants of interpretations using syntactic and semantic analysis, as described in this document.

[0168] in at least one embodiment of the implementation, when the user choose� variant interpretation he is returned.

[0169] in at least one embodiment of the implementation, the user is offered an interface to edit the interpretation before it is returned.

[0170] in at least one embodiment of the implementation, the user is offered an interface to proceed with additional voice input before the input is returned. This gives him the opportunity to incrementally compose the utterance input, receiving the syntactic and semantic correction, advice and guidance during one iteration.

[0171] in at least one embodiment of the implementation, the user is offered an interface to move directly to step 136 111 active way of extracting the printed input (described above in connection with Fig.11). This gives him the opportunity to punctuate the printed and spoken input, receiving the syntactic and semantic correction, advice and guidance at one stage.

[0172] in at least one embodiment of the implementation, the user is offered an interface to move directly from stage 111 of the embodiment of the active extract of the printed input to the embodiment of the active extract of the speech input. This gives him the opportunity to punctuate the printed and spoken input, receiving the syntactic and semantic korra�tion, advice and guidance on one stage.

Active extract input based on the GUI

[0173] Referring now to Fig.23 shows a block diagram of the sequence of operations illustrating a method for active retrieval of input for input based on the GUI according to one variant of implementation.

[0174] the Method 140 begins. Assistant 1002 is 141 graphical user interface (GUI) on the device 1207 output, which may include, for example, links or buttons. The user interacts 142, at least one GUI element. Data 144 are accepted and 146 is converted into a universal format. The converted data is then returned.

[0175] in at least one embodiment of the implementation, some GUI elements are generated dynamically from the active models of the ontology, and are not recorded in the computer program. For example, assistant 1002 can offer a set of constraints to guide the reservation service of the restaurant as areas to touch on the screen, and each area represents the name and/or value. For example, the screen may have a string of dynamically generated GUI layout with areas for restrictions by cuisine, location and price range. If the model is active ontology changed, the GUI screen should automatically reprogrammed to change without�I.

Active extract input recommendations on dialogue

[0176] Fig.24 is a block diagram of the sequence of operations illustrating a method for active extract input at the flow level dialogues according to one variant of implementation. Assistant 1002 recommends 151 answers 152. The user selects 154 recommended answer. Accept input 154 is converted into a universal format. The converted data is then returned.

[0177] in at least one embodiment of implementation, the recommendations proposed in step 151, offered as subsequent steps in the flow of dialogue and/or tasks.

[0178] in at least one embodiment of implementation, the recommendations suggest ways to Refine the query, for example, using the parameters from the domain model and/or tasks. For example, you may be prompted to change the approved location or the time of the request.

[0179] in at least one embodiment of implementation, the recommendations offer options to choose between ambiguous alternative interpretations provided through a process or component of language interpretation.

[0180] in at least one embodiment of implementation, the recommendations offer options to choose between ambiguous alternative interpretations, giving�generated through a process or component of language interpretation.

[0181] in at least one embodiment of implementation, the recommendations offer options to select between the following stages associated with the stream processing model 1087 flow of dialogues. For example, the model 1087 flow of dialogue may recommend that after the collection of constraints for a single subject area (e.g., dinner at a restaurant) assistant 1002 recommended other related subject area (e.g., the closest the film).

Active monitoring of relevant events

[0182] in at least one embodiment of the implementation, asynchronous events can be processed as inputs similar to other active modalities of recoverable input. Thus, such events can be provided as inputs to the assistant 1002. After interpretation, such events can be handled in a manner similar to any other type.

[0183] for Example, changing the status of a flight can initiate the sending of push notifications to the user. If the flight is indicated as lagging, assistant 1002 can continue the dialogue by presenting alternative flights, the formation of other recommendations, etc. on the basis of detected events.

[0184] Such events can be of any type. For example, assistant 1002 may detect that the user just came home or deviated (from the uke�data of the route), or what the stock price reaches the threshold value, or what starts the show interesting to the user, or that for a musician makes a tour in the area. In any of these cases, the assistant 1002 can continue the dialogue almost identical to as if the user initiates the request. In one embodiment, the implementation, the events may even be based on data available from other devices, for example, to tell the user when a colleague returned from lunch (device colleagues can send such an event to the user's device, and at this point of time, the assistant 1002 installed in the user's device and responds accordingly).

[0185] In one embodiment of implementation, the events may be notices or alerts from the calendar, the clock, the app works with reminders or with a list of classes. For example, a notification from the app to using the calendar on the date of the dinner can initiate dialogue with the assistant 1002 to meet at dinner in the restaurant. The dialogue may proceed as if the user just said or printed information regarding the upcoming dinner in the restaurant, for example, "dinner for two in San Francisco."

[0186] In one embodiment of implementation, the context of a possible trigger 162 events may�locate information about people, places, times and other data. These data can be used as part of the input in the assistant 1002 to be used at various stages of processing.

[0187] In one embodiment, the implementation, the data from the context of the trigger 162 events can be used to disambiguate speech input or inputs text from the user. For example, if an alert concerning the calendar event includes the name of the person invited to the event, this information can help to disambiguate the input, which can map multiple people with the same or similar name.

[0188] Referring now to Fig.25 shows a block diagram of the sequence of operations illustrating a method for active monitoring of relevant events according to one embodiment of the. In this example, the events trigger events are the sets of input 162. Assistant 1002 monitors on the subject of 161 such events. Detected events can be filtered and sorted 164 for semantic relevance using models, data and information available from other components of intelligent automated assistant 1002. For example, for an event that reports a change in the status of the flight may be granted a higher relevantist�, if the records are short-term or long-term storage device for the user indicate that the user is on that flight and/or executes a query about it in the assistant 1002. This sorting and filtering may then submit only the main events for review by the user who can then select one or more of them and influence them.

[0189] event Data 166 is converted into a universal format of input and returned.

[0190] in at least one embodiment of implementation, the assistant 1002 can offer the service associated with the events that are recommended for your attention. For example, if the alert status of the flight indicates that the flight can be canceled, assistant 1002 may recommend to the user the task to re-route planning or booking a hotel.

An example of the component(s) extraction type

[0191] the Following example has the intention to help illustrate some of the various types of functions, operations, actions, and/or other signs that may be provided by the component(s) 1094 active extract input.

Example: Fill commands (what the user can say to the assistant 1002?)

[0192] the User displays a text input field with the General instruction to enter "What would you do?" Depending on the�STI on context and user input is any one of several responses of the system. Examples are shown below.

Case: Zero input

Case: Enter the first word

User inputSystem responseComment
When empty dialog context, if the user types the first letter of any command, it is recommended. For example,
F|
- Find restaurants, hotels, movies
-
Black text is inserted in the input field. The italic text represents a further recommendation after the inserted text.

Case: Enter keywords

User inputSystem responseComment
When empty dialog context, if the user types the first two letters of any keywords, it is recommended as a team. For example,
Re|
- To recall the lesson,...
- Find restaurants by location, cuisine
-
Black text is inserted in the input field. Italic text repre�ulation of a further recommendation after as text is inserted.

Case: Prompt for arguments

User inputSystem responseComment
The user has entered or selected a "command" is a verb and nothing else.
For example,
Find_ | (Find)
- restaurants by location, cuisine
hotels by location, availability,...
movies by location, genre,...
- the name of the place
Besides the class of choice, but also issued advice on the options for restrictions. It should be noted that the name of the place is just a string with the prompt and should not paste the text.

Case: recommendation criteria

Case: Adding criteria

User inputSystem responseComment
The user has said "in" and you are now prompted as a tooltip on the item locations
Find restaurants in_ | (Find restaurants)
- city, postcode
- Palo Alto, California
- Menlo Park, California
Until he enters the next word, recommendation are there-a clue as to what to print. Places are cities from a personal storage device.
The user said "about" and you are now prompted as a tooltip on the item locations
Find restaurants near_ | (to Find restaurants around)
- my house
- my work
- Palo Alto, California
- Menlo Park, California
The places are one or more places from a personal storage device.

Case: add a location or other restrictions

Case: Beginning with restrictions, an unknown task or subject area

User
ski type
System responseComment
romantic_ | (romantic)- restaurants by location, cuisine,...
hotels by location, availability,...
movies by location, genre,...
-
Value constraints from the ontology tells You what classes of choice should be used.
comedy_ | (Comedy) movies by location, genre,...
- activities by location,...
clubs by location
"Comedy" is a constraint value for genre in movies, genre in the events, and "Comedy clubs" are a category of local directory organizations

Example: Filling in names

[0193] Here, the user has typed some text without confirming any of the teams, or is he just completes the command with the name of the object. The system can attempt to fill in names depending on the context. It also resolves the ambiguity of the subject area.

Case: words without context

User inputSystem responseComment
il for- Il Fornaio (restaurant)
- Ill Forgotten Gains (film)
May require searching on the object name. It should be noted verbose filling. The display subject region as a recommendation.
tom cruise (Tom cruise)moviesMay require searching on the object name.
romantic (romantic)- fil�s
- restaurants
Only on the basis of the ontology

Case: name with context

User inputSystem responseComment
Find restaurants il for (to Find restaurants il for)- Il Fornaio (restaurant)
-
May require searching on the object name. Uses a class of choice and the context of the location of the current and previous queries to limit the filling. The display subject region as a recommendation.
tom cruise (Tom cruise)moviesMay require searching on the object name.
Find movies rom (Find movies rom)- romantic
- Roman holiday (film)
Only on the basis of the ontology
g-rated (rated G)movies

Example: Selecting values from the set

[0194] Here, the user responds to the system a request to enter a value for a specific parameter, such as location, time, cuisine, or genre. The user can either in�the collection from the list either enter the value. As it prints, matching items from the list are displayed as options. Examples are shown below.

Case: select valueclass

User inputSystem responseComment
Restaurants serving (restaurantes)- burgers
- ice cream
- hot dogs
The user can print anything else and definitely don't have to choose from the menu.
Restaurants serving h (Restaurants offering h)- burgers
- hot dogs
- hot sauce
Movies playing Movies, running)- today
- tonight
- Friday night

Example: reusing previous commands

[0195] Previous queries are also options that should be filled in UI autocomplete. They can simply be mapped as a string (when the input field is blank, no known limitations), or they can be relevant in certain situations.

Case: completion of previous requests

Example: extract the elements of the personal storage device

[0196] the Assistant 1002 is able to remember certain events and/or objects in personal storage device associated with the user. Autocomplete can be made on the basis of such stored items. See the examples below.

Case: completion on events and objects in personal storage device

User inputSystem responseComment
Mary- lunch with Mary on Saturday (from My Events My events))
films called "Something about Mary"
Lunch (Lunch)- lunch with Mary on Saturday (from My Events My events))
- buy book Naked lunch (from My Todos (My classes))
HobRestaurant Hobee's in Palo Alto (from My Restaurants (My restaurants))

Active extraction of multimodal input

[0197] in at least one embodiment, the implementation component(s) 1094 active extract input can handle input innogest modalities of input. At least one modality can be implemented with the active extract input, which enjoys the advantages of particular types of inputs and how to select from the proposed matches. As described herein, it may be possible to implement procedures for the active retrieval of input for a text input, speech input, input, GUI, input, in the context of a dialogue and/or I, which is the result of trigger events.

[0198] in at least one embodiment of the implementation, for a single instance of intelligent automated assistant 1002 may be to support one or more (or any combination) of the printed input, speech input, GUI input dialog input and/or input events.

[0199] Referring now to Fig.26 shows a block diagram of the sequence of operations illustrating a method for active retrieval of multimodal input according to one variant of implementation. Method 100 begins. The inputs can be taken simultaneously from one or more or any combination of input modalities in any sequence. Thus, the method includes the active extract printable entry 2610, speech input 2620, 2640 input based on the GUI, input, 2650 in the context of a dialogue and/or input 2660, which is the result of triggers for events. Any or all� of these input sources unified in unified format 2690 input and return. A uniform format 2690 provides input to the establishment and operation of other components of intelligent automated assistant 1002 regardless of the specific modality of the input.

[0200] the active recommendation instructions for multiple modalities and levels provides a restriction and instructions for entering beyond the limitations and guidelines are available for stand-alone modalities. For example, the types of recommendations offered to choose between speech, text and dialogue, are independent, so that their combination is a significant improvement over the addition of the active extract in a separate modality or levels.

[0201] the Combination of multiple sources of constraints, as described in this document (syntactic/linguistic, dictionary, object database, domain models, models, tasks, models, services, etc.), and a few places in which these limitations can actively apply (this, text, GUI, dialogue and asynchronous events), provides a new level of functionality for human-computer interaction.

Component(s) 1056 domain models

[0202] Component(s) of the model 1056 subject areas includes representations of concepts, objects, relations, properties, and instances subject� area. For example, the model 1622 subject area dinners in restaurants may include the concept of the restaurant as an organization with the name and address and phone number, the concept of the event with the meal of the evening as well as date and time associated with the restaurant.

[0203] in at least one embodiment, the implementation component(s) 1056 domain models assistant 1002 may be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

Component(s) 1056 domain model can be used by automated assistant 1002 for several processes, including: extracting the input 100, the interpretation of natural language 200, the coordination of services for 400 and the formation o 600.

Component(s) 1056 domain model can provide a list of words that can match the concept of the subject area or object, for example, the names of the restaurants that can be used for active extract input 100 and natural language processing 200.

Component(s) 1056 domain model can classify the word choices in the processes, for example, to determine the �that word is the name of the restaurant.

Component(s) 1056 domain model can show the relationship between partial information for interpreting natural language, for example, that the kitchen can be associated with commercial organisations (for example, "local Mexican food" can be interpreted as "find restaurants with style= Mexican", and this logical conclusion is possible due to the information in the model 1056 subject area).

Component(s) 1056 domain model can organize information about services used in the 1082 orchestration services, for example, that a particular web service can provide reviews of restaurants.

Component(s) 1056 domain model can provide information for the formation of perefrazirovanie natural language and other formatting output, for example, by providing a canonical ways of describing concepts, relations, properties, and instances.

[0204] According to a specific implementation options, multiple instances or threads of the component(s) 1056 domain models can be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, at least in some vari�ntah implementation various aspects, features and/or functionality of the component(s) 1056 domain models can be performed, implemented and/or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes, etc. (or combinations of the above):

Component(s) 1056 domain models can be implemented as data structures that represent concepts, relations, properties, and instances. These data structures can be stored in the memory device, files or databases.

- Access component(s) 1056 domain model can be implemented through direct APIs, network APIs, interfaces, database queries, etc.

- Creation and maintenance of component(s) 1056 domain models can be implemented, for example, via direct editing of files, database transactions, and/or using the workbench editing domain models.

Component(s) 1056 domain models can be implemented as part of or in Association with active ontologies 1050 that combine models with implementations of models for servers and users.

[0205] According to different variants of implementation, one or more different threads or instances of the component(s) 1056 models subject �of areas can be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1056 domain models. For example, the initiation or start of implementation of one or more different threads or instances of the component(s) 1056 domain models can be initiated when required information of the domain model, including the extraction of inputs, input interpretation, identification of tasks and subject areas, natural language processing, orchestration of services and/or format the output for the user.

[0206] in at least one embodiment of the implementation, this instance of a component(s) 1056 domain models may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. For example, data of component(s) 1056 domain model can be associated with other components of the simulation-based models, including dictionary 1058, modules 1060 recognition language constructs, models 1087 flow of dialogue, model 1086 stream� tasks model 1088 characteristics of services, base 1072 data domain objects, etc., for Example in the databases 1072 data domain objects, which are classified as restaurants, can be known through the type identifiers that are supported in the components of the domain model dinners in restaurants.

An example of the component(s) of domain models

[0207] Referring now to Fig.27, shows a set of screen shots illustrating an example of various types of functions, operations, actions, and/or other signs that may be provided by the component(s) 1056 domain models according to one variant of implementation.

[0208] in at least one embodiment, the implementation component(s) 1056 domain models is a unifying representation of data that provides a view of information displayed on the screens 103A and 103B, the restaurant, which combines data from several different data sources and services and which includes, for example: name, address, categories, organizations, telephone number, ID to store in long-term personal storage device, the identifier for sharing by email, reviews from multiple sources, mapping coordinates, personal� notes, etc.

Component(s) 1070 language interpreter

[0209] in at least one embodiment, the implementation component(s) 1070 language interpreter assistant 1002 may be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

• Analysis of user input and to identify a set of grammatical analysis results.

- User input can include any information from the user and the context of a device which may contribute to the understanding of the user's intent, which may include, for example, one or more of the following (or a combination of the above): sequences of words, the identity of gestures or GUI elements that are involved in the extraction of the input current context of the dialogue, the current application of the device and its current data objects and/or any other personal dynamic data obtained about the user, such as location, time, etc. for Example, in one embodiment of the implementation, user input has the form of a universal format annotated 2690 input resulting 1094 active extract input.

- The results of grammatical analysis are asso�Yerevanian data in the user input with the concepts, interactions, properties, instances and/or other nodes and/or data structures in the models, databases and/or other representations of the intentions and context of the user. Association results of grammatical analysis can be a complicated transformations of sets and sequences of words, signals and other elements of the user input in one or more of the associated concepts, relations, properties, instances, other units and/or data structures described in this document.

• Analysis of user input and to identify a set of results of the parse, which are the results of grammatical analysis that associate data in the user input with the structures that represent the syntactic parts of speech, expressions and phrases, including long-winded title, the structure of sentences and/or other grammatical structure of the graph. The results of the parse are described in item 212 of the procedure of natural language processing described in connection with Fig.28.

• Analysis of user input and to identify a set of results of semantic analysis, which are the results of grammatical analysis that associate data in the user input with the structures that represent concepts, relations, St.�an, objects, values, judgments, and/or other representations of the values and intentions of the user. In one embodiment of the implementation, these values and intentions are represented by sets and/or elements and/or model instances or databases, and/or nodes in the ontology, as described in item 220 of the procedure of natural language processing described in connection with Fig.28.

• Disambiguation between alternative outcomes syntactic or semantic analysis, as described in item 230 of the procedure of natural language processing described in connection with Fig.28.

• Determining whether or not partially printed input syntactically and/or semantically meaningful in the procedure autocomplete, for example, in the procedure described in connection with Fig.11.

• Assistance in the formation of the recommended padding 114 in procedure autocomplete, for example, in the procedure described in connection with Fig.11.

• Determining whether or not the interpretation of the spoken input is syntactically and/or semantically meaningful in the procedure speech input, for example, in the procedure described in connection with Fig.22.

[0210] According to a specific implementation options, multiple instances or threads of the component(s) 1070 language interpreter may be concurrently implemented and/or initiated via the ISP�Lituanie one or more processors 63 and/or other combinations of hardware and/or hardware and software.

[0211] According to different variants of implementation, one or more different threads or instances of the component(s) 1070 language interpreter may be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1070 language interpreter. Various examples of conditions or events that may initiate or start the implementation of one or more different threads or instances of the component(s) 1070 language interpreter, may include, but not limited to, one or more of the following (or a combination of the above):

• when removing the entry, including, but not limited to:

- the recommendation of possible fillings printable input 114 (Fig.11);

- ranking of interpretations of the speech 126 (Fig.22);

- upon the recommendation of the ambiguities as recommended by the answers in the dialogue 152 (Fig.24);

• when the result of the extraction of input available, including when the input is removed via any mode of active extract 100 multimodal input.

[0212] in at least one embodiment of the implementation, this instance of a component(s) 1070 language interpreter may access and/or use the information �with one or more associated databases. In at least one embodiment, the implementation of at least part of such information, the database access may be via a connection to one or more local and/or remote storage devices. Examples of different types of data that can be accessed through the component(s) of a language interpreter, may include, but not limited to, one or more of the following (or a combination of the above):

model 1056 subject areas.

- dictionary 1058;

- base 1072 data domain objects;

- short-term storage device 1052;

- long-term personal storage device 1054;

model 1086 flow problems;

model 1087 stream dialogues;

model 1088 characteristics of services.

[0213] referring now to Fig.29 shows a screen shot illustrating a natural language processing according to one variant of implementation. The user provides (via voice or text) language input 2902, consisting of the phrase "who's playing this weekend at filmore". This phrase reflects the user on the screen 2901. Component(s) 1070 language interpreter component-by-component processes input 2902 and generates the result of grammatical analysis. The result of grammatical analysis associating the input with the request to display the local event to�e assigned to any of the upcoming weekends in any venue, which name coincides with the "filmore". Rephrasing the results of grammatical analysis as shown 2903 on the screen 2901.

[0214] Also referring now to Fig.28 shows the block diagram of the sequence of operations of a method illustrating an example of a method for natural language processing according to one variant of implementation.

[0215] Method 200 begins. Accepted language input 202, such as the line "Who's playing this weekend at filmore" in the example of Fig.29. In one embodiment, the implementation, the input is expanded by means of the current context information such as the user's current location and local time. When comparing 210 words/phrases, component(s) 1070 language interpreter finds Association between user input and concepts. In this example, the Association between "plays" and the concept of lists in the venues; the string "this weekend" (along with the current local time of the user), and a specialization of the approximated period of time, which is the coming weekend; and the string "filmore" with the name of the venue. Comparison of 210 words/phrases may use the data, for example, module 1060 recognition of linguistic structures, lexical databases 1058 data, active ontology 1050, short-term personal-only memory�th device 1052 and long-term personal storage device 1054.

[0216] the Component(s) 1070 language interpreter generates options 212 parses that include the selected result of grammatical analysis, but can also include other results of grammatical analysis. For example, other results of grammatical analysis may include results in which "plays" associated with other subject areas, for example, games, or category of events, such as sporting events.

[0217] the short and/or long-term storage device 1052, 1054 can also be used by the component(s) 1070 language interpreter in the formation of variants 212 parsing. Thus, the input of which is given earlier in the same session, and/or known information about the user can be used to improve performance, reduce ambiguity and enhance conversational nature of the interaction. Data from the active ontology 1050, model 1056 subject areas and models of 1086 task flow can also be used to implement based on the rationale of the reasoning in the determination of allowable options 212 parsing.

[0218] for semantic harmonization 220, the component(s) 1070 language interpreter considers Combe�nation possible results of grammatical analysis according to how well they correspond to semantic models, for example, domain models and databases. In this case, grammatical analysis includes associating (1) "playing" (the word in the user input) as a "local event at the venue (part of the model 1056 subject area, represented by a cluster of nodes in the active ontology 1050) and (2) "filmore" (another word in the input) as a coincidence with the name of the object in the database 1072 data domain objects to venue for local events, which is represented by the element of the domain model and the active node of the ontology (name of venue).

[0219] the Semantic harmonization 220 may use data from, for example, active ontology 1050, short-term personal storage device 1052 and long-term personal storage device 1054. For example, semantic harmonization 220 may use data from previous references to the venue or local events in the dialogue (from short-term personal storage device 1052) or favorite venue (from long-term personal storage device 1054).

[0220] the Set of results (or potential�outcomes) semantic analysis 222 is formed.

[0221] In step 230 of ambiguity resolution, component(s) 1070 language interpreter weights based on the rationale of the intensity variants 222 results of semantic analysis. In this example, the combination of grammatical analysis "plays" as a local event in the venue and match "filmore" as the name of the venue is more strict coincidence with the domain model than alternative combinations in which, for example, "plays" associated with the domain model for sports, but no Association in the subject area of sports for "filmore".

[0222] the disambiguation 230 may use the data, for example, from the structure of the active ontology 1050. In at least one embodiment of the implementation, the connections between nodes in the active ontology are granted based on justification support to resolve the ambiguity between 222 results of semantic analysis. For example, in one embodiment, the implementation, if three nodes active ontology is semantically the same and all are connected in the active ontology 1050, it indicates higher based on the rationale of the intensity of the semantic analysis than when these coincident nodes are not connected or connected through�Twomey longer join paths in the active ontology 1050. For example, in one embodiment, the implementation of semantic matching 220, for grammatical analysis, which coincides with both "local event venue" and "the name of the venue is enhanced based on the rationale support, because the combined views of these aspects of the user's intention are connected through links and/or relationships in active ontology 1050: in this case, the host local events connected to the host venue, which is connected with the node name of the venue, which is connected with the name of the object in a database of the names of the venues.

[0223] in at least one embodiment of the implementation, the connections between nodes in the active ontology, based on which provide the rationale for support to resolve the ambiguity between 222 results of semantic analysis are directed arcs that form the lattice of logical inference, in which coincident nodes provide a rationale for the nodes to which they are connected by directed arcs.

[0224] In step 232 component(s) 1070 language interpreter sorts and selects 232 main variations of semantic analysis as a nama submission� " s 290 of the user.

Database(s) 1072 data domain objects

[0225] in at least one embodiment of implementation, the database(s) 1072 data domain objects can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

• Save the data about the domain objects. The domain objects are entities in the world or in a computing environment that can be modeled in the domain models. Examples may include, but not limited to, one or more of the following (or a combination of the above):

- organization of any kind;

movies, videos, songs and/or other musical products and/or any other entertainment products by name;

- products of any kind;

- event;

- calendar entries;

- cities, States, countries, around, and/or other geographical, geopolitical and/or geospatial point or region;

- a place called, for example, points of interest, airports, etc.;

• Provision of services to work with databases these databases, including but not limited to simple and complex queries, transactions, trigger events, etc.

[0226] According to a particular variation�there implementation multiple instances or threads of the base 1072 data domain objects may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the base 1072 data domain objects may be performed, implemented and/or initiated via the database software and/or hardware, permanently reside on the client(s) 1304 and/or the server(s) 1340.

[0227] in One example, base 1072 data domain objects, which can be used in connection with the present invention according to one variant of implementation, is the database one or more organizations that preserve, for example, their names and location. The database can be used, for example, to search for words contained in the input request, on the subject of matching organizations, and/or identifying the location of the organization, the name of which is known. Specialists in the art should recognize that many other configurations and implementations are possible.

Component(s) 1058 dictionary

[0228] At least � one variant of implementation, component(s) 1058 dictionary can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- Providing a database associating words and strings of concepts, properties, relations, or instances of domain models or models of tasks;

- Vocabulary of the components of the dictionary can be used by automated assistant 1002 for several processes, including, for example: extracting input, interpretation of natural language and the formation of the output.

[0229] According to a specific implementation options, multiple instances or threads of the component(s) 1058 dictionary may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component(s) 1058 dictionary can be implemented as data structures that associate the strings with the names of concepts, relations, properties, and instances. These data structures can be stored in a storage device�ve, files or databases. Access to the component(s) 1058 dictionary can be implemented through direct APIs, network APIs and/or interfaces, database queries. Creation and maintenance of component(s) 1058 dictionary can be made via direct editing of files, database transactions, or using the workbench editing domain models. Component(s) 1058 dictionary can be implemented as part of or in Association with the active ontology 1050. Specialists in the art should recognize that many other configurations and implementations are possible.

[0230] According to different variants of implementation, one or more different threads or instances of the component(s) 1058 dictionary can be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1058 dictionary. In one embodiment, the implementation, access to the component(s) 1058 dictionary can be whenever requested, dictionary information, including, for example, in extracting input, interpret, input and formatting output for users. Specialists in the art should recognize that other conditions or in the event�I can initiate or start the implementation of one or more different threads or instances of the component(s) 1058 dictionary.

[0231] in at least one embodiment of the implementation, this instance of a component 1058 dictionary may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. In one embodiment, the implementation component 1058 dictionary can access data from external databases, for example, from the data repository or the dictionary.

Component(s) 1060 recognition module language constructs

[0232] in at least one embodiment of the implementation, the component 1060 recognition module language constructs can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics such as, for example, a search of structures in language or speech input, which indicate grammatical, idiomatic, and/or other composite object markers type. These structures correspond to, for example, one or more of the following (or combinations of the above): words, titles, phrases, data, parameters, commands and/or signals of speech acts.

[0233] According to a specific implementation options, multiple instances or bounce or move to�processes component(s) 1060 recognition module designs may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware tools and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component(s) 1060 recognition module language constructs can be performed, implemented and/or initiated by one or more files, databases and/or programs containing expressions in the language matching design. In at least one embodiment, the implementation component(s) 1060 recognition module language constructs represented declaratively, rather than as code; this allows them to create and maintain by editors and other tools than programming tools. Examples of declarative representations may include, but not limited to, one or more of the following (or a combination of the above): regular expression matching rules with the design, the grammar of a natural language, grammatical analyzers based on finite state machines and/or other models of grammatical analysis.

[0234] the Specialists in the art should recognize that other types of systems, components, systems, devices, procedures, processes, etc. (or combinations of the above) can be used to implement the component(s) 1060 module� recognition language constructs.

[0235] According to different variants of implementation, one or more different threads or instances of the component(s) 1060 recognition module language constructs can be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1060 recognition module language constructs. Various examples of conditions or events that may initiate or start the implementation of one or more different threads or instances of the component(s) 1060 recognition module language constructs may include, but not limited to, one or more of the following (or a combination of the above):

- during the active extract of the entry in which the structure of the recognition engines are language constructs can limit and direct the input from the user;

- during the processing of the natural language in which the recognition engines are language constructs help to interpret the input as a language;

- in the course of identification of tasks and dialogues in which the recognition engines are language constructs can help to identify tasks and/or milestones.

[0236] in at least one embodiment of the implementation, this instance to�component(s) 1060 recognition module language constructs may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be accessed through the component 1060 recognition module language constructs may include, but not only, data from any of the models, different models and data sources that may be part of embodiments of assistant 1002, which may include, but not limited to, one or more of the following (or a combination of the above):

model 1056 subject areas.

- dictionary 1058;

- base 1072 data domain objects;

- short-term storage device 1052;

- long-term personal storage device 1054;

model 1086 flow problems;

model 1087 stream dialogues;

model 1088 characteristics of services.

[0237] In one embodiment of the implementation, access data from other parts of embodiments of assistant 1002 may be coordinated through the active ontology 1050.

[0238] referring Again to Fig.14 shows an example of some different types of functions, operations, actions, and/or other signs that may be provided by the component(s) 1060 module recognizing�of language constructs. Fig.14 illustrates the language constructs that identify a component(s) 1060 recognition module language constructs. For example, the idiom "what's happening" (in the city) can be associated with the task of organizing events and subject area local events.

Component(s) 1080 processor flow dialogs

[0239] in at least one embodiment, the implementation component(s) 1080 processor of the flow of the dialogues can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- Considering the representation of intentions 290 user of language interpretation 200, identification of tasks, performance of which requires the user, and/or problems, resolution of which requires the user. For example, the task may be to find a restaurant.

- For the given problem or task, taking into account the submission of 290 intentions of the user, identification of parameters for a task or problem. For example, the user can search for a recommended restaurant that serves Italian food around the house user. Such restrictions as what a restaurant should be recommended, the supply of Italian food and around the house are settings for the task at�of ardenia restaurant.

- Taking into account the interpretation of the task and an ongoing dialogue with the user, for example, which can be represented in short-term personal storage device 1052, the choice of the appropriate model of the flow of dialogue and the definition of stages in the model of the thread corresponding to the current state.

[0240] According to a specific implementation options, multiple instances or threads of the component(s) 1080 processor thread of the conversations may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software.

[0241] in at least one embodiment of the implementation, this instance of a component(s) 1080 processor flow of dialogue may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be accessed through the component(s) 1080 processor thread conversations that may include, but not limited to, one or more of the following (or a combination of the above):

model 1086 sweat�ka tasks;

model 1056 subject areas.

model 1087 flow dialogs.

[0242] Referring now to Fig.30 and 31, shown screen shots illustrating an example of various types of functions, operations, actions, and/or other signs that may be provided by the component(s) CPU thread dialogues according to one variant of implementation.

[0243] As shown in the screen 3001, the user requests a reservation for dinner by providing voice input or text input "Book me a table for dinner." Assistant 1002 provides a string 3003 with a prompt requesting that the user has specified the time and the number of participants soiree.

[0244] Once these parameters are provided, shown screen 3101. Assistant 1002 displays a dialog box 3102, indicating that the results are presented, and the line 3103 with a prompt asking the user to click. Lists 3104 are also displayed.

[0245] In one embodiment of the implementation, such a dialogue is implemented as follows. In component(s) 1080 processor flow of dialogue is a representation of a user's intention from the component 1070 language interpreter, and he determines what the appropriate response is to ask the user for the information required in order to perform the next stage in poto�e tasks. In this case, the subject area are restaurants, the task is redundant, and the dialogue stage is to ask the user for the information required in order to perform the next step in the task flow. This phase of the dialogue is illustrated by means of a line 3003 with a prompt on the screen 3001.

[0246] Also referring now to Fig.32 shows the block diagram of the sequence of operations illustrating a method of operation for the component(s) 1080 processor of the flow of the dialogues according to one variant of implementation. A block diagram of the sequence of operations of the method of Fig.32 described in connection with the example shown in Fig.30 and 31.

[0247] Method 200 begins. Submission intentions 290 user is accepted. As described in connection with Fig.28, in one embodiment, the implementation, the representation of intentions 290 user is a set of semantic analysis. For example, shown in Fig.30 and 31, the subject area are restaurants, the verb is "to order" associated with the reservation of the restaurant, and the time setting is the evening of the current day.

[0248] In step 310, the component(s) 1080 processor of the flow of the dialogues is supported or not this interpretation of the intention of the user is quite strict, to continue and/or supported it or not better alternative prices. e�tive ambiguous variants of semantic analysis. In the present example, the interpretation is supported strictly without competing ambiguous variants of semantic analysis. On the other hand, if there are competing ambiguity or sufficient uncertainty, then step 322 is performed to set the stage of a stream of dialogues, so that the execution phase instructs the dialog to display a string with a prompt for additional information from the user.

[0249] In step 312, the component(s) 1080 processor flow dialogs defines the preferred interpretation of the semantic analysis with other information to determine the task that should be performed and its parameters. Information can be obtained, for example, model 1056 subject areas, models 1086 flow of tasks and/or models 1087 flow of dialogue, or any combination thereof. In the present example, the task is identified as a backup which contains as finding a place that is available for reservation and free, and the implementation of the transaction to reserve a table. The parameters of the problem are a temporary restriction together with others that are logically derived in step 312.

[0250] In step 320 apply to the model of task flow in order to determine the appropriate next stage. Information can be obtained, for example, from model�th 1056 subject areas, models 1086 flow of tasks and/or models 1087 flow of dialogue, or any combination thereof. In the example, it is determined that the task flow the next step is to extract the missing options to search for the availability of restaurants, leading to line 3003 with tip illustrated in Fig.30, requesting the number of participants in the evening and time for backup.

[0251] As described above, Fig.31 illustrates the screen 3101, which is shown as including an element 3102 dialogue that appears after the user responds to the query regarding the number of participants soiree and backup time. In one embodiment, the implementation, the screen 3101 is represented as the result of another iteration through the procedure of the automated call and response, as described in connection with Fig.33, which leads to another procedure call conversations and threads, illustrated in Fig.32. In this specification procedures dialogues and flows, after taking user preferences component(s) 1080 processor flow of dialogue defines a different stage of the workflow at step 320: to search for availability. When a request 390 is compiled, it includes the parameters of the problem, sufficient for 1080 component processor's flow of dialogues and component(s) 1082 orchestration services�, to be coordinated for the service of orders of the restaurant.

Component(s) 1087 models of the flow of the dialogues

[0252] in at least one embodiment, the implementation component(s) 1087 models of the flow of the dialogues can be arranged to provide flow model dialogues that represent the steps that are being taken in a particular kind of conversation between the user and the intelligent automated assistant 1002. For example, the flow of the dialogues for the generalized problem of the transaction includes the steps to obtain the necessary data for the transaction and confirm the settings of the transaction prior to its conclusion.

Component(s) 1086 models of task flow

[0253] in at least one embodiment, the implementation component(s) 1086 models of task flow can be configured to provide flow model tasks that represent the steps that are being taken to resolve the problem or satisfy the need. For example, the task flow for a dinner reservation encompasses finding the desired restaurant, the availability check and the transaction to get a reservation for a specific time in the restaurant.

[0254] According to a specific implementation options, multiple instances or threads of the component(s) 1086 models of the flow of tasks can be simultaneously R�lyzovani and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component(s) 1086 models of task flow can be implemented as software, state machines or other ways of identifying the appropriate stage in the block diagram.

[0255] in at least one embodiment, the implementation component(s) 1086 models of task flow can use the infrastructure of the modeling of tasks, called a generalized tasks. Generalized tasks are abstractions that model the stages in the problem and their required inputs and generated conclusions without depending on subject areas. For example, generalized problem for a transaction may include the procedures for collecting the data required for the transaction, transaction and output the results of the transaction - and all this without regard to a particular subject area transaction or service for its implementation. It may be specific to the subject area, such as shopping, but it is independent of the subject area purchases and can with equal success be applied to subject areas reservation, scheduling, etc.

[0256] At least a part of functions, operations, actions, and/or other attributes associated with the component(s) 1086 models of the flow of tasks and/or procedure�Roy(AMI), described in this document can be implemented, at least partially, with the use of concepts, features, components, processes, and/or other aspects disclosed in this document in respect of infrastructure modeling generalized tasks.

[0257] Additionally, at least some of the functions, operations, actions, and/or other attributes associated with the component(s) 1086 models of the flow of tasks and/or procedure(s) described herein may be implemented, at least partially, with the use of concepts, features, components, processes, and/or other aspects associated with the task selection with restrictions as described in this document. For example, one variant of implementation of generalized tasks can be implemented using a model of choice with constraints.

[0258] in at least one embodiment of the implementation, this instance of a component(s) 1086 flow models task may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be access through�m component(s) 1086 models of task flow, may include but not limited to, one or more of the following (or a combination of the above):

model 1056 subject areas.

- dictionary 1058;

- base 1072 data domain objects;

- short-term storage device 1052;

- long-term personal storage device 1054;

model 1087 stream dialogues;

model 1088 characteristics of services.

[0259] Referring now to Fig.34 shows a block diagram of the sequence of operations of a method illustrating an example of a workflow for the task 351 of choice with constraints according to one variant of implementation.

[0260] the Choice of constraints is a form of the generalized problem in which the goal is to select an element from a set of items in the world on the basis of a set of constraints. For example, the task 351 selection restrictions can be specified for the subject area restaurants. Task 351 choice with limitations begins by querying criteria and constraints from the user 352. For example, the user may be interested in Asian food and want a place to eat, near his office.

[0261] In step 353, the assistant 1002 represents the elements that satisfy the criteria and limitations for the user to view them. In this example, it can be a list of re�Taranov and their properties, which can be used to choose one of them.

[0262] In step 354, the user is given the opportunity to clarify the criteria and constraints. For example, the user can Refine the query by means of expressing "near my office. The system then needs to introduce a new set of results in step 353.

[0263] Also referring now to Fig.35, shown is an example screen 3501, which includes a list 3502 elements represented by the problem 351 of choice with constraints according to one variant of implementation.

[0264] In step 355, the user can choose from the matching elements. Can then be provided access to any number of future tasks 359, such as, for example, ordering 356, reminder 357 or sharing 358. In various embodiments, further tasks 359 may have major interactions with the services based on web technologies and/or with the functionality local to the device (for example, setting appointments on the calendar, a phone call, send email or text message, setting an alarm, etc.).

[0265] In the example of Fig.35, the user can select an item in the list 3502 to see additional information and perform additional actions. Also referring now to Fig.36, shown is an example screen 3601 after custom�l chose an item from the list 3502. Additional information and options appropriate to further the objectives 359 relative to the selected item are displayed.

[0266] In various embodiments, the steps of flow may be offered to the user in any of several input modalities, including but not limited to any combination of explicit dialog lines with a prompt and a GUI-references.

Component(s) 1084 services

[0267] the Component(s) 1084 service represents the set of services that an intelligent automated assistant 1002 may cause on behalf of the user. Any service that can be invoked, can be offered in the component services 1084.

[0268] in at least one embodiment, the implementation component(s) 1084 service can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- Provide API functions that would normally be provided through web user interface service. For example, review the web-site can provide the services API, which automatically returns the reviews of this object when calling through the program. API offers intelligent automated assistant 1002 services that people otherwise $ �Yong to get through the work with the user interface of our web site.

- Provide API functions that normally must be provided by the user interface in the app. For example, an application for working with calendar can provide the services API, which automatically returns the calendar entry when calling through the program. API offers intelligent automated assistant 1002 services that people would otherwise have to obtain by working with the user interface of the application. In one embodiment, the implementation, the assistant 1002 has the ability to initiate and control any number of different functions available in the device. For example, if the assistant 1002 is installed on the smartphone, personal digital device, a tablet computer, or other device, assistant 1002 can perform functions such as: launching apps, making calls, sending emails and/or text messages, add calendar events, setting alarms, etc., In one embodiment, the implementation of such functions are activated by using the component(s) 1084 services.

- Providing services that are not currently implemented in the user interface, but are available through the API helper on big tasks. For example, in one embodiment, the implementation, the API to receive the actual �adresses and return the machine-readable geo-coordinates can be used by assistant 1002 as a component of 1084 service, even if it has no direct user interface based on web technology or device.

[0269] According to a specific implementation options, multiple instances or threads of the component(s) 1084 services may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component 1084 service may be performed, implemented and/or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes, etc. (or combinations of the above):

- implementation of the API provided through the service, either locally or remotely in any combination;

- the implication of a database in an automated assistant 1002 or service database available to the assistant 1002.

[0270] for Example, a web site that offers users an interface for watching movies, can be used through an embodiment of intelligent automated assistant 1002 as a copy of the database used by the web-site. Component(s) 1084 service then must propose inside�tions API for data as if they are on a network API, even if the data is stored locally.

[0271] In another example, component(s) 1084 services for intelligent automated assistant 1002, which helps with restaurant and event organization with eating, and may include any or all of the next set of services that are available from third parties over the network:

- a set of services providing the list of restaurants that list the restaurants that match by name, location or other restrictions;

- a set of services ranking restaurants that return a ranking for restaurants by name;

- a set of services review restaurants that return written reviews for restaurants by name;

service geocoding to locate restaurants on a map;

reservation service that provides programmable booking tables in restaurants.

Component(s) 1082 orchestration services

[0272] the Component(s) 1082 orchestration services intelligent automated assistant 1002 completes the orchestration of services.

[0273] in at least one embodiment, the implementation component(s) 1082 orchestration services can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or each�x signs, such as, for example, one or more of the following (or a combination of the above):

- Dynamically and automatically determining which service can satisfy the user request and/or specify the subject area(s) and purpose(s).

- Dynamic and automatic invocation of multiple services, in any combination of parallel and sequential ordering;

- Dynamic and automatic conversion of parameters and constraints in the problem to meet the entry requirements, the services API;

- Dynamic and automatic tracking and collecting results from multiple services;

- Dynamic and automatic Association of data with the provision of services from various services in a unified model results;

- The orchestration of multiple services to meet the constraints of the query;

- The orchestration of multiple services to annotate an existing set of results with supporting information;

- Display the result of calling multiple services in a universal, independent from the service view, which unifies the results from different services (for example, as the result of invoking multiple services of the restaurant, which return a list of restaurants, integrate data, at least one restaurant from multiple services, removing redundancy).

[0274] �of primer, in some cases, there may be several ways to perform a specific task. For example, user input, for example, "remind me to go to a meeting in the city at 14:00" indicates an action that can be performed in at least three ways: to set the alarm; create a calendar event; or call the Manager of the list of classes. In one embodiment, the implementation component 1082 orchestration services performs the determination as to which method is best suited for the query.

[0275] the Component(s) 1082 orchestration services can also perform determination regarding which combination of multiple services it is best to activate to perform this common task. For example, to find and to reserve a table for dinner, component(s) 1082 orchestration services must comply with the definition regarding what services should be called to perform functions such as searching in the reviews, obtaining availability and redundancy is performed. Determining which service to use may depend on any number of different factors. For example, in at least one embodiment of the implementation, information about the reliability of that service's ability to handle certain types of queries, feedback to the user, etc. can be used as a factor in ODA�division, what service(s) must be activated.

[0276] According to a specific implementation options, multiple instances or threads of the component(s) 1082 orchestration services can be concurrently implemented and/or initiated via the use of one or more processors and/or other combinations of hardware and/or hardware and software.

[0277] in at least one embodiment of the implementation, this instance of a component(s) 1082 orchestration services can use an explicit model 1088 characteristics of the services to represent characteristics and other properties of external services, and to reason about these characteristics and properties when you reach the signs of the component(s) 1082 orchestration services. This provides advantages compared to manual coding of the set of services that may include, for example, one or more of the following (or a combination of the above):

- ease of development;

- stability and reliable performance;

- the ability to dynamically add and remove services without damaging the code;

- the ability to realize total optimization algorithms for distributed queries, which are controlled by the properties and characteristics instead of hard-coded for particular services or API.

[0278] in at least one embodiment implemented�of tvline, this instance of the component(s) 1082 orchestration services may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. Examples of different types of data that can be accessed through the component(s) 1082 orchestration services that may include, but not limited to, one or more of the following (or a combination of the above):

- specification of domain models;

- options syntax and semantic analysis of the input in a natural language;

- specification of task models (with values for the parameters);

- flow model dialogues and tasks and/or selected stages in them;

model 1088 characteristics of the services;

- any other information available in the active ontology 1050.

[0279] Referring now to Fig.37, shown is an example procedure to perform the procedure orchestration of services according to one variant of implementation.

[0280] In this particular example, it is assumed that one user is interested in finding good places for dinner in the restaurant and in contact with intelligent automated assistant 1002 in the form R�of shoora, to help provide these services.

[0281] Consider the problem of finding restaurants that have high quality, good reviews in the reviews are about specific locations available for reservation at a particular time and offer specific kind of food. These parameters of the subject area and objectives are provided as input 390.

[0282] Method 400 begins. In step 402 it is determined whether or not the request to require any service. In some cases, delegation services may not be required if, for example, assistant 1002 has the ability to perform the required task on their own. For example, in one embodiment, the implementation, the assistant 1002 may be able to answer the actual question without activation of delegation services. Accordingly, if the request does not require service, the offline phase flow is performed in step 403, and 490 is returned. For example, if the request for the task is to request information about the automated assistant 1002, the response of the dialogue can be handled without activating external services.

[0283] If in step 402 it is determined that delegation is required, the services component(s) 1082 orchestration services continues to step 404. In step 404 component(s) 1082 orchestration services can match task requirements decl have been designed with�operational descriptions of the characteristics and properties of services in models 1088 characteristics of services. At least one service provider that can support the specified operation, provides a declarative, high-quality metadata, detailing, for example, one or more of the following (or a combination of the above):

- data fields that are returned with the results;

- what classes of options service provider (statically known) supports;

functions of policy management for the parameters, which the service provider can support after dynamic checking of parameter values;

- performance rating that specifies how the service works (for example, relational DB, web service, stored in three copies, the full-text index or some combination of the above);

- ratings of quality properties, statically defining the expected quality of the property values returned by object result;

- a rating of the overall quality of the results that can be expected to return the service.

[0284] for Example, when considering classes of parameters that can support the service, the services model may indicate that the services 1, 2, 3 and 4 can provide restaurants that are near a specific location (parameter), services 2 and 3 can filter or rank restaurants according to the quality (another parameter), services 3, 4 and 5 may return review�s for restaurants (returned data field), service 6 may list the types of meals offered by restaurants (a data field returns), and service 7 can check the availability of restaurants in specific time ranges (option). Service 8-99 offer features that are not required for this particular domain and task.

[0285] using these declarative, high-quality metadata, tasks, task parameters, and other information, is available from the runtime helper component(s) 1082 orchestration services 404 determines an optimal set of service providers that you want to activate. The optimal set of service providers may support one or more parameters of the problem (it returns results that satisfy one or more parameters), and also considers a rating performance of at least one service provider and the rating of the overall quality of at least one service provider.

[0286] the Result of step 404 is a dynamically generated list of services that should call for this particular user and the request.

[0287] in at least one embodiment, the implementation component(s) 1082 orchestration services considers the reliability of services and their ability to respond to specific information requests.

[0288] in at least one embodiment implemented�I, component(s) 1082 orchestration services confronts the unreliability by calling overlapping or redundant services.

[0289] in at least one embodiment, the implementation component(s) 1082 orchestration services treats personal information about the user (from a component of short-term personal storage device) to select the service. For example, the user may prefer some services for the rankings over the other.

[0290] In step 450, the component(s) 1082 orchestration services dynamically and automatically activates several services on behalf of the user. In at least one embodiment of the implementation, they are invoked dynamically in response to the user request. According to specific variants of implementation, can be called multiple instances or threads of the services. In at least one embodiment of the implementation, they are invoked by the network using the API or via the network using APIs web services, or via the Internet using web API services, or for any combination of the above.

[0291] in at least one embodiment of implementation, the frequency with which the called service, software limited and/or controlled.

[0292] Also referring now to Fig.38, shown is an example of procedure 450 service activation according to one embodiment osushestvlenie�. The service activation is used, for example, to obtain additional information or perform tasks through the use of external services. In one embodiment of implementation, the query parameters are converted properly for service API. Once the results are accepted from the service, the results are converted to the presentation of results for presentation to the user in the assistant 1002.

[0293] in at least one embodiment of implementation, the service is activated through a process 450 service activation may be a web service application running on the device, the function of the operating system, etc.

[0294] Provided representation 390 of the request, including, for example, task parameters, etc. for At least one service available from models 1088 characteristics of services, procedure 450 service performs the conversion steps 452, call 454 and 456 conversion output.

[0295] In step 452, the current conversion task options from the view 390 query is converted into a form that can be used by at least one service. Options for services that may be offered as an API or database may be different from the data representation used in queries by task and also at least from each other. Accordingly, the purpose of stage 42 is to convert the at least one task parameter in one or more respective formats and values, at least one of the called service.

[0296] for Example, names of organizations, for example, restaurants may vary according to the services, which consider such organizations. Accordingly, step 452 must include the conversion of any names in the forms that are best suited, at least for the same service.

[0297] In another example, the known location at different accuracy levels and using different units and symbols according to the services. Service may 1, may require zip codes, service 2 - GPS-coordinates, and 3 - the actual postal addresses.

[0298] the Service is invoked 454 API, and its data is collected. In at least one embodiment of implementation, the results are cached. In at least one embodiment, the implementation of the services that are not returned within the specified performance level (e.g., as shown in the agreement on service levels or SLA) are discarded.

[0299] In step 456 conversion output, the data returned by the service is converted back into a unified representation 490 result. This phase may include consideration of different formats, units�, etc.

[0300] In step 412, the results from multiple services checked for validity and unite. In one embodiment, the implementation, if the validated results are compiled, the function of policy management equality (specified for each subject area) is then called in pairs for one or more results to determine which results are identical concepts in the real world. When the detected pair of equal results, the set of functions of policy management properties (also set for each subject area) is used to combine the values of the properties in the merged result. The function of policy management properties may use the ratings of the quality properties of the models characteristics of services, settings, tasks, context of the subject area and/or long-term personal storage device 1054 to determine the optimal join strategy.

[0301] for Example, a list of restaurants from different vendors can be combined and duplication removed. In at least one embodiment of implementation, criteria for the identification of overlaps may include fuzzy matching of names, fuzzy matching locations, fuzzy matching with multiple properties of domain objects, such as name, month�polozhenie, phone number and/or address of the web site and/or any combination thereof.

[0302] In step 414, the results are sorted and cut to return a list of results desired length.

[0303] in at least one embodiment of implementation, the contour of the weakening of the request also applies. If at step 416 component(s) 1082 orchestration services determines that the current list of results is not sufficient (for example, he has less than the required number of matching elements), the parameters of the problem may diminish 420 to provide a larger number of results. For example, if the number of restaurants required type found within N miles from the target location, is too small, the attenuation must run the query again with the search field larger than N miles, and/or weakening of some other search parameter.

[0304] in at least one embodiment of implementation, the method for orchestration of services is applied in the second pass to "annotate" the results with supporting data that are useful for the task.

[0305] In step 418 component(s) 1082 orchestration services determines whether or not the annotation. It may be required if, for example, the task may require the plan with the results on the map, but the primary service does not return the geographical coordinates required for p�of obrazovanie.

[0306] In step 422 appeal again to the model 1088 characteristics of services in order to find services that can return the additional information required. In one embodiment, the implementation, the process of annotating determine who can or not additional or better data to be annotated in the merged result. He does this by delegating to the function of policy management properties (as specified for each subject area), at least one property of at least one merged result. The function of policy management properties can use the combined property value and rating of the quality characteristics, quality ratings of properties of one or more other service providers, the context of the subject area and/or user profile to determine what may or not be the very best data. If it is determined that one or more service providers can annotate one or more properties for the combined result, activated by a cost function to determine an optimal set of service providers that you want to annotate.

[0307] At least one service provider in the optimal set of annotated service providers can then be activated at 450 with the list of merged results to get the results 424. The changes made by IU�Isha least in one combined result by at least one provider of services that are monitored during this process and the changes are then combined using the process of the function of policy management properties identical to that used in step 412. Their results are combined 426 to the existing set of results.

[0308] the Resulting data are sorted 428 and unified in a universal representation 490.

[0309] it is Possible to take into account that one advantage of the methods and systems described above relative to the component(s) 1082 orchestration services is that they can mostly be applied and/or used in various fields of engineering that are different from the fields of technology, which, in particular, are associated with intelligent automated assistants. Examples of such other areas of technology in which aspects and/or features of procedures of orchestration of services that include, for example, one or more of the following:

- dynamic "mixing" on web sites and web applications and services;

- optimization of distributed database queries).

- dynamic configuration of service-oriented architecture.

Component(s) 1088 models characteristics of services

[0310] in at least one embodiment, the implementation component(s) 1088 models the characteristics of the services can be vypolnen opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- Providing machine-readable information about the characteristics of the service to execute certain classes of computations;

- Providing machine-readable information about the characteristics of the services to answer certain classes of queries.

- Providing machine-readable information about what classes of transactions are provided through various services;

- Providing machine-readable information about the parameters for the API presented by the various services;

- Providing machine-readable information about the parameters that can be used in database queries to databases is provided through various services.

Component(s) 1090 processor output

[0311] in at least one embodiment, the implementation component(s) 1090 processor output can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- The formatting of the output data, which is presented as a universal internal structure of the data in the forms and models that prepare them properly �La different modalities. The output data may include, for example, communication in natural language between intelligent automated assistant and the user; data domain objects, for example, properties, restaurants, movies, products, etc.; the results depend on the subject with the field data from information services, for example, weather forecasts, check flight status, pictures, etc.; and/or interactive links and buttons that enable the user to respond via a direct interaction with the representation of the output.

- Preparing output for modalities, which may include, for example, any combination of the following: graphical user interfaces; text messages; email messages; sounds; animations; and/or speech output.

- Dynamic preparation of data for various display mechanisms on the basis of graphical user interface based on the query. For example, the use of different layouts and formats output processing depending on which web browser and/or device being used.

- Dynamic preparations output in a variety of voices in the form of the speech signal.

- Dynamic preparations for the specified modalities based on user preferences.

- Dynamic preparations in�water using specific user "shells", that set design and functionality.

- Send in the modality of the output packet stream, showing an intermediate state, feedback or results on all phases of interaction with the assistant 1002.

[0312] According to a specific implementation options, multiple instances or threads of the component(s) 1090 processor output may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component(s) 1090 processor output may be performed, implemented and/or initiated by one or more of the following types of systems, components, systems, devices, procedures, processes, etc. (or combinations of the above):

- software modules in a client / server embodiment of intelligent automated assistant;

- remotely callable service;

- use a combination of templates and procedural code.

[0313] Referring now to Fig.39 shows a block diagram of the sequence of operations of a method illustrating an example procedure of the multi-phase output according to one variant of implementation.The procedure of multiphase output includes stages 702 processing in an automated assistant 1002 and stages 704 multiphase output.

[0314] In step 710 turns out the pronunciation of the speech input, and component convert your speech to text (for example, the feature described in connection with Fig.22) interprets it to form the set of options 712 interpretation of speech. In one embodiment, the implementation, the component converting speech to text is implemented using, for example, recognition module nuances offered by company Nuance Communications, Inc., Burlington, MA. Options 712 interpretation of the speech can be shown to the user at 730, for example, in paraphrased form. For example, the interface may display alternatives "did You say?", listing several possible alternative interpretations of one text sample speech sound.

[0315] in at least one embodiment, the implementation is provided a user interface to allow the user to pause and select from options for the interpretation of speech.

[0316] In step 714 options 712 interpretation of the speech is sent in the language interpreter 1070, which can form representations of intentions 716 user at least one option for 712 interpretation of speech. In step 732, rephrasing these representations intentions 716 user are generated and presented to the user. (Cm. the associated stage 132 of procedure 120 in Fig.22).

[0317] in at least one variation�the implementation the user interface allows the user to interrupt and to choose from perefrazirovanie interpretations 732 in natural language.

[0318] In step 718 is performed analysis tasks and dialogues. In step 734 the interpretation of the task and the subject area are presented to the user using the algorithm of rephrasing intentions.

[0319] Also referring now to Fig.40 shows a screen shot illustrating an example of output processing according to one variant of implementation. Screen 4001 includes a reflection 4002 speech input of the user information generated by stage 730. Screen 4001 additionally includes paraphrasing 4003 intention of the user, is formed through the step 734. In one embodiment of the implementation, as illustrated in the example of Fig.40 for keywords such as "activities", there is a special formatting/highlighting, which can be used to simplify the learning user to interact with intelligent automated assistant 1002. For example, by visual observation of the formatting of the displayed text, the user can easily identify and interpret back that intelligent automated assistant recognizes keywords, such as "events", "in the following�following Wednesday," "San Francisco", etc.

[0320] referring Back to Fig.39, as requests are coordinated 720 for services, and the results dynamically gather the intermediate results can be displayed in the form of stroke 736 run in real time. For example, you might get a list of restaurants, and then the reviews can be dynamically populated as do the results of service reviews. The service may include service on the basis of web-technologies and/or services that access information stored locally in the device, and/or from any other source.

[0321] a Universal representation of the response 722 is formed and formatted 724 for proper modality of output. After the final output formatting is completed, another kind of paraphrasing may be offered in step 738. In this phase, the entire set of results can be analyzed and compared with the initial request. Then you may be offered a summary of the results or the answer to the question.

[0322] Also referring to Fig.41 shows another example of output processing according to one variant of implementation. Screen 4101 illustrates the paraphrasing 4102 interpretation of the text formed by the step 732, the course 4103 run in real time, formed by step 736, and paraphrased summary 7104 formed through�Twomey stage 738. Also included detailed results 4105.

[0323] In one embodiment of implementation, the assistant 1002 allows the formation of a conclusion in several modes. Referring now to Fig.42 shows a block diagram of the sequence of operations of a method illustrating an example of processing multimodal output according to one variant of implementation.

[0324] the Method begins in step 600. The processor 1090 o adopt universal response representing 490 and formats 612 answer according to the device and modality that is appropriate and applicable. Step 612 may include information from the device and models 610 modality and/or models 614 the subject area.

[0325] After the response 490 formatted 612, any number of different inference mechanisms can be used in any combination. The examples illustrated in Fig.42, include:

- Formation of 620 in text messages that is sent 630 to the channel text messages;

- Formation of 622 o e-mail 632 is sent as a mail message.

- Formation of 624 GUI output that is sent 634 in the device or web browser to prepare;

- Formation 626 speech output that is sent 636 to the module of formation of speech.

[Package 0326] Specialists in the art should priznava�ü, what can be used many other inference mechanisms.

[0327] In one embodiment, the implementation, the content of output messages generated by the procedure 700 multiphase output adapts to the treatment regimen of 600 multimodal output. For example, if the output modality is speech 626, the language used to paraphrase the user input 730, interpretation 732 text, interpretation 734 task and the subject area, the course 736 execution and/or summary of 738 results may be more or less verbose, or use the suggestions that are easier to detect in audio form than in written form. In one embodiment, the implementation, the language adapts to the stages of the procedure 700 multiphase output; in other embodiments, the procedure 700 multiphase output leads to an intermediate result, which is further specified in a particular language by processing 600 multimodal output.

Component(s) 1052 short-term personal

storage devices

[0328] in at least one embodiment, the implementation component 1052 short-term personal storage device can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other signs, still� as, for example, one or more of the following (or a combination of the above):

- Storage background of the recent dialogue between the variant of implementation of the assistant and the user, including the history of user inputs and their interpretations;

- Storage background of the recent elections by the user in the GUI, for example, what elements are open or understood, what phone numbers called, what are the elements that transformed what movie trailers played, etc.;

- Save history of dialogue and user interaction in the database on the client, the server-specific user session or in the state of the client session, such as a cookie of the web browser or RAM used by the client;

- Save list of recent user requests;

- Maintaining consistency between the results of the recent user queries;

- Save the prehistory of the sequence of clicks UI events, including button clicks, touches, gestures, triggers the activation of voice and/or any other user input.

- Save data from the device sensors (such as location, time, position, orientation, motion, light level, sound level, etc.) that can be correlated with interactions with the assistant.

[0329] With�to their specific implementation options, multiple instances or threads of the component(s) 1052 short-term personal storage device may be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software.

[0330] According to different variants of implementation, one or more different threads or instances of the component(s) 1052 short-term personal storage device may be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1052 short-term personal storage device. For example, component(s) 1052 short-term personal storage device can be activated when there is a user session with a variant implementation of assistant 1002, in at least one form of input or action by the user, or a response through the system.

[0331] in at least one embodiment of the implementation, this instance of a component(s) 1052 short-term personal storage device may access and/or use information from one Il� more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices. For example, component(s) 1052 short-term personal storage device can access data from a component(s) 1054 long-term personal storage device (for example, to obtain user identification data and personal settings) and/or data from a local device with respect to time and location, which may be included in the record of short-term storage device.

[0332] Referring now to Fig.43A and 43B, shown screen shots illustrating an example of using the component(s) 1052 short-term personal storage device to maintain a dialog context of changing the location, according to one embodiment of the. In this example, the user asked about local weather, and then just says "new York". The screen shows 4301 initial response, including local weather. When the user says "new York", assistant 1002 uses the component(a) 1052 short-term personal storage device to access the dialog con�extu and thereby define the what current subject area are forecasts. This provides an opportunity to the assistant 1002 to interpret a new articulation of the "new York" as meaning "what is the weather forecast in new York this coming Tuesday?". Screen 4302 shows a proper response, which includes the weather forecasts for new York.

[0333] In the example of Fig.43A and 43B, in short-term storage device saved not only the words you enter "the day after Tomorrow it will rain?", but the semantic interpretation of the input system as the subject area of the weather and the time parameter defined as the day after tomorrow.

Component(s) 1054 long-term personal

storage devices

[0334] in at least one embodiment, the implementation component 1054 long-term personal storage device can be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (or a combination of the above):

- Permanent retention of personal information and data about the user, including, for example, his preferences, identification information, authentication credentials, accounts, addresses, etc.;

- Save the information of the user collected through the use of in�Rianta implementation of assistant 1002, such as the equivalent of bookmarks, favorites, samples, etc.;

- Permanent preservation of the saved list of commercial organizations, including restaurants, hotels, shops, theatres and other venues. In one embodiment, the implementation component(s) 1054 long-term personal storage device stores more than just the name or URL, and saves the information, sufficient to cause a complete list of objects, including phone number, map location, photos, etc.;

- Permanent preservation of stored movies, videos, music, shows and other entertainment elements;

- Permanent preservation of a personal calendar(s), list(cov) practice, reminders and alerts, databases, contacts, lists, social network user, etc.;

Permanent saving shopping lists and wish lists for products and services received coupons and discount codes, etc.;

- Permanent preservation of history and receipts transactions, including reservations, purchase event tickets, etc.

[0335] According to a specific implementation options, multiple instances or threads of the component(s) 1054 long-term personal storage device can be implemented simultaneously �/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. For example, in at least some embodiments, implementation, various aspects, features and/or functionality of the component(s) 1054 long-term personal storage device may be performed, implemented and/or initiated using one or more databases and/or files to (or associated) clients 1304 and/or servers 1340 and/or permanently reside in the storage device.

[0336] According to different variants of implementation, one or more different threads or instances of the component(s) 1054 long-term personal storage device may be initiated in response to detection of one or more conditions or events satisfying one or more different types of minimum threshold criteria for triggering at least one instance of a component(s) 1054 long-term personal storage device. Various examples of conditions or events that may initiate or start the implementation of one or more different threads or instances of the component(s) 1054 long-term personal storage device, may include, but not limited to, one or more of the following (or a combination of the above):

Record long-term personal storage device can be detected�wife as a side effect of user interaction with a variant implementation of assistant 1002. Any kind of interaction with the assistant can generate add to long-term personal storage device, including browsing, searching, finding, purchasing, scheduling, acquisition, redundancy, data exchange with other people through the assistant.

- Long-term personal storage device may also be filled as a result of user registration in the account or service, ensure that the assistant 1002 access to accounts in other services, use the assistant 1002 service on the client device with access to other databases of personal information, such as calendars, lists of classes, contact lists, etc.

[0337] in at least one embodiment of the implementation, this instance of a component(s) 1054 long-term personal storage device may access and/or utilize information from one or more associated databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote storage devices, which may be, for example, on the client(s) 1304 and/or the server(s) 1340. Examples of different types of data that can be access through�ohms component(s) 1054 long-term personal storage device, may include but not limited to data from other databases of personal information, such as contact lists or buddy lists, calendars, lists of classes, other dispatchers lists, dispatchers personal accounts and wallets provided by external services 1360, etc.

[0338] Referring now to Fig.44A-44C, shown screen shots illustrating an example of using the component(s) 1054 long-term personal storage device, according to one embodiment of the. In the example, is a sign (called "My Stuff (My stuff)"), which includes access to stored objects, such as restaurants, movies, and organizations that are using interactive sessions with a variant implementation of assistant 1002. On the screen 4401 of Fig.44A, the user has found the restaurant. The user touches the Save to My Stuff (to Keep in My supplies) 4402, which stores information about the restaurant in the component(s) 1054 long-term personal storage device.

[0339] the Screen 4403 of Fig.44B illustrates a user accessing My Stuff (My stuff). In one embodiment, the implementation, the user can select from the categories to navigate to the desired element.

[0340] the Screen 4404 of Fig.44C illustrates My Restaurant category (My restaurants), including elements previously save�record in My Stuff (My stuff).

The procedure of the automated call and response

[0341] Referring now to Fig.33 shows a block diagram of the sequence of operations of a method illustrating a process of automatic call and response according to one embodiment of the. Procedure of Fig.33 may be implemented in connection with one or more embodiments of intelligent automated assistant 1002. It is possible to take into account that intelligent automated assistant 1002, as illustrated in Fig.1 is just one example from a wide range of options for implementing the system based on the intelligent automated assistant, which can be implemented. Other embodiments of the systems based on the intelligent automated assistant (not shown) may include additional, fewer and/or other components/signs compared to those that are illustrated, for example, in an exemplary intelligent automated assistant 1002, shown in Fig.1.

[0342] in at least one embodiment of the implementation, the procedure of the automated call and response of Fig.33 may be performed with the opportunity to perform and/or implement various types of functions, operations, actions, and/or other characteristics, such as, for example, one or more of the following (and�and combinations of the above):

- The procedure of the automated call and response of Fig.33 may provide a flow path management interface for conversational interface between the user and the intelligent automated assistant 1002. At least one iteration of procedure of the automated call and response can serve as a branch in the conversation. A conversational interface is the interface in which the user and the assistant 1002 to communicate by alternately reciting phrases in a conversational tone.

- The procedure of the automated call and response of Fig.33 can provide Executive control flow for intelligent automated assistant 1002. In other words, the procedure controls the collection of input, processing input, the formation of a conclusion and presenting output to the user.

- The procedure of the automated call and response of Fig.33 may coordinate communication between the components of intelligent automated assistant 1002. In other words, she can direct where the output of one component is fed to another, and may occur where the total input from the environment and the effect on the environment.

[0343] in at least some embodiments, the implementation of part of the procedure of the automated call and response can also be implemented in other devices and/or systems�x computer network.

[0344] According to a specific implementation options, multiple instances or threads of procedure of the automated call and response can be concurrently implemented and/or initiated via the use of one or more processors 63 and/or other combinations of hardware and/or hardware and software. In at least one embodiment of the implementation, one or more or the selected part of the procedure of the automated call and response can be implemented in one or more client(s) 1304, in one or more server(s) 1340, and/or in combination thereof.

[0345] for Example, in at least some embodiments, implementation, various aspects, features and/or functionality of the procedure of the automated call and response can be performed, implemented and/or initiated via software components, network services, databases, etc. or any combination thereof.

[0346] According to different variants of implementation, one or more different threads or instances of the automated procedure call and response can be initiated in response to detection of one or more conditions or events satisfying one or more different types of criteria (such as, for example, minimum threshold criteria) for triggering, by �ENISA least one instance of procedure of the automated call and response. Examples of different types of conditions or events that may initiate or start the implementation of one or more different threads or instances of procedure of the automated call and response may include, but not limited to, one or more of the following (or a combination of the above):

• custom session with an intelligent automated assistant 1002, such as, for example, but not limited to, one or more of the following:

- launch app for mobile devices, such as applications for mobile devices, which is an implementation of the embodiment of intelligent automated assistant 1002;

- launching computer applications, such as applications, which is an implementation of the embodiment of intelligent automated assistant 1002;

- a dedicated button on a mobile device, for example, the speech input button";

- the button on a peripheral device attached to a computer or a mobile device such as a headset, handset or base station, a GPS navigation system, household appliance, remote control or any other device with a button that can be associated with a list�her care;

- starting web session from the web browser to a web site, implementing an intelligent automated assistant 1002;

- the beginning of the interaction from an existing session of a web browser to a web site, implementing an intelligent automated assistant 1002, in which, for example, the requested service intelligent automated assistant 1002;

- sending mail (e-mail) messages to the server 1426 modality that mediates communication with a variant implementation of intelligent automated assistant 1002;

- send a text message to the server 1426 modality that mediates communication with a variant implementation of intelligent automated assistant 1002;

- making a call to the server 1434 modality that mediates communication with a variant implementation of intelligent automated assistant 1002;

- sending event such as an alert or a notification in the app that provides variant implementation of intelligent automated assistant 1002.

• when a device that provides intelligent automated assistant 1002 is enabled and/or running.

[0347] According to different variants of implementation, one or more different threads or instances of the procedure automatiser�vanny call and response can be initiated and/or implemented manually, automatically, statically, dynamically, concurrently, and/or combinations thereof. Additionally, different instances and/or embodiments of the procedure of the automated call and response can be initiated in one or more different time intervals (e.g., during a specific time interval, at regular periodic intervals, at irregular periodic intervals, upon demand, etc.).

[0348] in at least one embodiment of the implementation, this instance of the procedure of the automated call and response can be used and/or form various special types of data and/or other types of information when performing specific tasks and/or operations. They may include, for example, the input data/information and/or output data/information. For example, in at least one embodiment, the implementation of at least one instance of the procedure of the automated call and response may access, process, and/or otherwise utilize information from one or more different types of sources, such as, for example, one or more databases. In at least one embodiment, the implementation of at least part of the database information may be accessed via communication with one or more local and/or remote W�commemorating devices. Additionally, at least one instance of the procedure of the automated call and response can form one or more different types of output data/information, which, for example, can be stored in a local storage device and/or remote storage devices.

[0349] in at least one embodiment of implementation, the initial configuration of this instance of the procedure of the automated call and response can be performed using one or more different types of initialization parameters. In at least one embodiment, the implementation of at least part of initialization parameters can be accessed via communication with one or more local and/or remote storage devices. In at least one embodiment, the implementation of at least a portion of the initialization parameters provided to an instance of procedure of the automated call and response, may correspond to and/or can be removed from the input data/information.

[0350] In a specific example of Fig.33, it is assumed that one user accesses the instance of intelligent automated assistant 1002 via the network from a client application with the characteristics of speech input. The user is interested in finding good places for dinner in the restaurant � contact with intelligent automated assistant 1002 in the form of a conversation, to help provide these services.

[0351] In step 100, the user is queried as the prompt to enter a query. The client UI offers several input modes as described in connection with Fig.26. They may include, for example:

- interface to the printed input that can activate the active procedure of extracting the printed input, as illustrated in Fig.11;

interface for speech input that can activate the active procedure of extraction of the speech input, as illustrated in Fig.22.

- interface to select entries from the menu, which can activate the active extract input based on the GUI, as illustrated in Fig.23.

[0352] the Specialists in the art should recognize that may be provided to the other input modes.

[0353] In one embodiment of implementation, the stage 100 may include a presentation of options remaining from the previous conversation with assistant 1002, for example, using the techniques described in the procedure active extract input recommendations for dialogue, described in connection with Fig.24.

[0354] for Example, using one of the methods of active extract input in step 100, the user can say to the assistant 1002: "Where can I find good Italian food around here?" For example, the user may p�iznosi in this component of the speech input. Variant implementation of the component 1094 active extract input causes the service to convert speech to text, prompts the user for confirmation and then is confirmed the user input as a universal format annotated 2690 input.

[0355] embodiment of the component 1070 language interpreter is then called in step 200, as described in connection with Fig.29. Component 1070 language interpreter grammatically analyzes text input and generates a list of possible interpretations of the intentions 290 user. In one grammatical analysis of the word "Italian" associated with restaurants in the Italian style; the "good" associated with the property dining recommendations, and "in the area" associated with the parameter of the provisions describing the distance readings from a global sensor (e.g., the user's location provided by GPS on the mobile device).

[0356] In step 300, the representation of intentions 290 user is passed to the processor 1080 flow of dialogue, which implements a variant of the procedure of the analysis of conversations and threads, as described in connection with Fig.32. The processor 1080 stream dialogs determines the kind of interpretation of intention is most likely converts this interpretation in instances of domain models � the model parameters of the tasks and determines the next phase flow in the flow of conversations. In the present example, the domain model of the restaurants is specified using the task selection with constraints to find a restaurant by means of restrictions (style cuisine, the level of recommendation and the limitations of proximity). A flow model of the dialogues indicates that the next step is to get some examples of restaurants that satisfy these constraints, and present them to the user.

[0357] In step 400 variant of the procedure 400 orchestration flows and services to be activated via component 1082 orchestration services. It activates a set of services 1084 on behalf of a user request to find a restaurant. In one embodiment, the implementation of these services 1084 contribute some data to the overall result. Their data are combined, and the resulting list of restaurants appears to be universal, independent from the service form.

[0358] In step 500, the processor 1092 output generates a summary of dialogue outcomes, for example: "I found some recommended Italian restaurants nearby". Processor 1092 output combines this summary with data output and then sends the combination to the module that formats the output for a specific mobile device user in step 600.

[0359] In step 700, this specific device for the output packet is sent to the mobile device, clientele software in the device prepares it on the screen (or other output device) of the mobile device.

[0360] the User views this view and decided to explore various options. If the user has finished 790, the method terminates. If the user has not finished 490, another iteration loop is initiated by returning to step 100.

[0361] the Procedure of automatic call and answer can be used, for example, to the user's query "how about Mexican?". Such input may be extracted in step 100. In step 200, the input is interpreted as the restaurants in Mexican style and is combined with another condition (stored in short-term personal storage device 1052) to support the interpretation of the identical intentions, like last time, with one change in the parameter style of the restaurant. In step 300, this is a renewed intention forms the clarification of the request that is provided in component(s) 1082 orchestration services in step 400.

[0362] In step 400, the updated coordinated request for multiple services 1084, leading to a new set of restaurants, which is summarized in the dialogue on the stage 500, formatted for the device 600 and sent over the network to show new content on a user's mobile device in step 700.

[0363] In this case, the user finds a favorite restaurant, shows it on a map and sends instructions how to get there, friend.

[036] the Specialists in the art should recognize, what are the various options for implementation of procedure of the automated call and response (not shown) may include additional features and/or operation than that illustrated in the specific embodiment, the implementation of Fig.33, and/or may omit at least a portion of the attributes and/or operations procedures of the automated call and response, illustrated in a specific embodiment, the implementation of Fig.33.

Selection with constraints

[0365] In one embodiment, the implementation of intelligent automated assistant 1002 uses the selection restrictions in their interactions with the user, in order better to identify and provide elements that are likely of interest to the user.

[0366] the Choice of constraints is a form of generalized tasks. Generalized tasks are abstractions that characterize the types of domain objects, inputs, conclusions, and the control flow that are common to the class of problems. Selection problem with constraints is performed by selecting items from a set of choices of domain objects (e.g., restaurants) constraint-based selection (for example, desired cuisine or location). In one embodiment, the implementation, the assistant helps the user 1002 �sucit the space of possible choices, extracting constraints and user preferences, view options, and recommendation of actions to be performed for these choices, for example, to reserve, to buy, to remember or to share them. The task is completed when the user selects one or more elements for which to perform the action.

[0367] the Choice of constraints is useful in many contexts: for example, select movie to watch, dinner restaurants, a hotel for the night, places to buy books, etc. In General, the choice of constraints is useful when a known category, and is required to choose a category instance with some desired properties.

[0368] One traditional approach to the choice of restrictions is the service catalog. The user selects a category, and the system offers a list of choices. In the local directory, you can restrict the directory location, such as a city. For example, in the service of the yellow pages to select a book for the city and then looking for the category, and the book shows one or more items for this category. The main problem when using the service catalog is that the number of possibly relevant choices is great (for example, restaurants in this city).

[0369] Other� traditional approach is the application of database management, which provides a way to form a set of choices through the extraction request from the user, extract the matching elements and presentation elements in some fashion, which highlights the characteristic features. The user browses the rows and columns of the result set, possibly sorting the results or modifying the request until, until it finds a number of suitable options. The problem is when you use the service database is that it may require that the user operationalisierung their human need as a formal request and abstract machine used for sorting, filtering, and viewing to examine the resulting data. It is difficult for most people, even with the help of graphical user interfaces.

[0370] the Third conventional approach is to open search, for example, "local search". Searching is simple in implementation, but there are several problems with search engines, making it difficult to people the pursuit of choice with constraints. In particular:

- Same as directory search, user can't just enter a category and to consider one or more possible choices, but should narrow down the list.

- If the user can narrow the selection by limiting�values, it is unclear what constraints can be used (for example, you can look for places that are at a distance that you can walk, or open late?")

It is unclear how to specify the constraint condition (for example, indicated cuisine or type of restaurant and what are the possible values?)

- The conflict of multiple preferences; usually no objectively "best" answer to this case (for example, I need a place that is close and is cheap, offering delicacies with excellent service, and is open until midnight).

- Preferences are relative and they depend on what is available. For example, if the user can achieve a table in the restaurant with a very high rating, he can choose him, even if he is expensive. However, in General, the user prefers less expensive options.

[0371] In various embodiments, the assistant 1002 of the present invention helps to simplify the selection problem with constraints. In various embodiments, the assistant 1002 uses service database and search service, as well as other functionality, to reduce effort on the part of the user to the expression of what he is looking for, given what is available, and the choice of a satisfactory solution.

[0372] In various embodiments, p�effect, assistant 1002 can help simplify the selection restrictions for people from any number of different ways.

[0373] for Example, in one embodiment, the implementation, the assistant 1002 can operationalize properties in the constraint. The user specifies what he wants, from the point of view of the properties desired result. Assistant 1002 operationaliseren this input to formal restrictions. For example, instead of saying "find one or more restaurants less than 2 miles from downtown Palo Alto, kitchen which includes Italian food", the user can simply say "Italian restaurants in Palo Alto". Assistant 1002 may also operationalize the quality requested by the user that are not the settings for the database. For example, if the user requests romantic restaurants, the system can operationalize it as a text search or limitations in the form of matching tags. Thus, the assistant 1002 helps to overcome some of the problems that users might otherwise have during the selection with restrictions. The user easier to imagine and describe a satisfactory solution than to describe conditions that would identify the appropriate solution from the wrong decisions.

[0374] In one embodiment of implementation, the assistant 1002 can offer useful criter�and choice, and the user needs only to indicate what criteria are important now. For example, assistant 1002 may ask: "what has the value: price (cheaper is better), location (closer is better) rank (higher rank is better)?" Assistant 1002 can also recommend criteria that may require specific values; for example: "You can tell what cuisine You prefer or what dish You prefer.

[0375] In one embodiment of implementation, the assistant 1002 can help the user to decide between options that differ on a number of competing criteria (such as price, quality, availability and convenience).

[0376] By providing such instructions, the assistant 1002 can help users in making multi-attribute decisions in any one of several ways:

One is to reduce the dimension of the space, combining the raw data, for example, ratings from multiple sources into a composite quantitative indicator. Composite quantitative index can take into account knowledge of the problem domain of data sources (e.g., Zagat ratings can provide more accurate predictions of quality than Yelp).

- Another approach is to focus on under�abore criteria turning the problem "what are all the possible criteria for consideration and how they are combined?" in the selection of the most important criteria in this case (e.g., "which one is more important, price or proximity?").

Another way to simplify a decision is to prevent the default preferences and practices (for example, all entities are equal, with a higher rating, as well as the closer and cheaper the better). The system can also remember previous user responses that indicate their default values and preferences.

- Fourthly, the system can offer the characteristic properties of elements in the set of options that are not mentioned in the original query. For example, the user may queries the local Italian food. The system can offer a set of choices of restaurants and with them, a list of popular tags used by reviewers, or key phrase from the guide (for example, "a nice place for a date" "great pasta"). This can allow people to select a particular element and to perform a task. Studies show that the majority of people makes decisions by evaluating specific instances instead of selecting criteria and best confirmation for the one that appears first. They also show �about, what people will learn about the characteristics of specific cases. For example, when choosing vehicles, buyers can not pay attention to the navigation system as long as they do not see that some cars have them (and subsequently, the navigation system may be an important criterion). Assistant 1002 may represent the characteristic properties of these elements that help people to choose the winner or recommended the dimension along which to optimize.

The conceptual data model

[0377] In one embodiment of implementation, the assistant 1002 offers assistance with the selection problem with constraints, through the simplification of the conceptual data model. The conceptual data model is an abstraction presented to the users in the interface of assistant 1002. To overcome the psychological problems described above, in one embodiment, the implementation assistant 1002 provides a model that enables users to describe what they want from the point of view of a few easily recognizable and memorable properties suitable options of choice, not expressions of limitation. Thus, properties can be simple when composing queries in natural language (e.g., adjectives, modifying the handles keyword) and can be recognized in rows on�tale ("You can also opt for recommended restaurants..."). In one embodiment, the implementation uses a data model that allows the assistant 1002 to specify the subject area (e.g. restaurants compared to hotels) and the General approach to instruction that can be fleshed out with depending on the subject area properties.

[0378] In one embodiment of implementation, the conceptual data model used by assistant 1002, includes a class of choice. It is the space representation of entities from which to choose. For example, the application "find a restaurant", the selection class is a class of restaurants. The selection class can be abstract and have subclasses, such as "classes at destination". In one embodiment, the implementation, the conceptual data model assumes that in this situation the solution, the user is interested in selecting one class of choice. This assumption simplifies the interaction and also allows the assistant 1002 to announce their boundaries of competence ("I know about restaurants, hotels and movies" as opposed to "I know about life in the city").

[0379] With regard to class selection, in one embodiment, the implementation of the data model presented to the user for selection of the restrictions include, for example: elements; the element indices; selection criteria; and limitations.

[0380] e�cops are instances of a class choice.

[0381] the Signs of the elements are properties, attributes or calculated values that can be represented and/or associated at least with one element. For example, the name and phone number of the restaurant are the signs of the elements. The signs can be internal (the name or the restaurant's kitchen) or relational (e.g., distance from current location for). They can be static (e.g., the name of the restaurant) or dynamic (a rating). They can be composite values calculated from other data (for example, a quantitative indicator of the ratio of price and quality"). The signs of the elements are abstractions for the user, performed by means of modeling the domain; they should not match the underlying data from a front-end services.

[0382] the selection Criteria are the signs of elements that can be used to compare the value or relevance of items. In other words, it is the ways to tell which elements are preferred. Selection criteria are modeled as attributes of the elements, regardless of whether they are internal properties or calculated. For example, proximity (defined as the distance from the interested location) is Cree�eriam choice. Location in space and time is a property, not a selection criterion, and it is used together with of the location to calculate the distance from the interested location.

[0383] the selection Criteria may have inherent order of preference. In other words, the value of any specific criterion can be used to align elements in order according to the criterion of best fit. For example, the criterion of proximity is inherent preference that closer is better. The location, on the other hand, has no inherent value preference. This restriction allows the system to make assumptions on default and direct the selection, if the user only mentions the criterion. For example, the user interface may offer a "sort by rating" and to assume that the higher ranking the better.

[0384] One or more selection criteria are also signs of the elements; these are signs associated with the choice of possible elements. Nevertheless, the signs of the elements are not necessarily associated with a preference (e.g., names and phone numbers of restaurants are usually not relevant for the choice of them).

[0385] in at least one embodiment of implementation, constraints are restrictions on the required values of the selection criteria. FD�mum, constraints can be represented as membership in the set (e.g., type of cuisine includes Italian), coincident with a design (for example, the text of the review of the restaurant includes "romantic"), fuzzy inequality (for example, the distance is less than a few miles), the quality threshold value (for example, with a high rating) or more complex functions (for example, a good ratio of price and quality). To sufficiently simplify the situation for ordinary people, this model reduces data, at least one or more restrictions to character values that can be mapped as a word. Time and distance may be excluded from this reduction. In one embodiment, the implementation, operators and thresholds that are used to implement constraints are hidden from the user. For example, the restriction on the selection criteria, called "kitchen", can be represented as a character value, for example, "Italian" or "Chinese". The constraint on the rating is "recommended" (binary choice). For time and distance, in one embodiment, the implementation assistant 1002 uses their own views that handle a range of input and value restrictions. For example, the distance may be the distance that you can walk", and the time can be "tonight"; in one embodiment about�of westline, assistant 1002 uses special processing to match the more accurate input data.

[0386] in at least one embodiment of the implementation, some constraints can be binding constraints. This means that the task simply cannot be completed without this data. For example, it is difficult to choose a restaurant without some idea of the desired location, even if you know the name.

[0387] to summarize, the subject area is modeled as a selection of classes with signs of elements that are important to users. Some functions are used to select and order items available to the user - these signs are called selection criteria. Restrictions are symbolic constraints on selection criteria to narrow the set of elements to elements that are the same.

[0388] Often, several criteria can compete, and limitations may partially coincide. The data model reduces the task of choosing from the optimization problem (finding the best solution) to the problem of mapping (finding items that are well-suited for a set of given criteria and match the character set restrictions). Algorithms for selection criteria and constraints and determine the ordering described in the next section.

�the technology of choice with constraints

[0389] In one embodiment of implementation, the assistant 1002 performs the selection with constraints by considering as input an ordered list of criteria with implicit or explicit limitations for at least one and generate a set of variants of elements with characteristic features. Computationally, the task of choosing can be characterized as a nested search: first, the identification of the class of choice, then the identification of the important selection criteria, then the specification of constraints (boundaries of feasible solutions) and perform a search in the instances in order of the best approximation to find an acceptable elements.

[0390] Referring now to Fig.45, shown is an example of an abstract model 4500 for the problem of selection restrictions as a nested search. In the example, assistant 1002 identifies 4505 challenge of choice for all types of local search 4501. Identified class is the restaurant. In the set of all the restaurants 4502, assistant 1002 selects 4506 criteria. In the example, the criterion is identified as distance. In the set of restaurants in PA 4503, assistant 1002 indicates 4507 restrictions for searching. In the example, the identified limitation is "Italian cuisine". In the set of Italian restaurants in PA 4504, assistant 4508 selects elements for presentation to the user.

[0391] In one embodiment, be implemented thr�tion, such a nested search is that assistant 1002 performs when he has the relevant input data, not the stream to retrieve the data and presentation of results. In one embodiment, the implementation of such control flow is managed through a dialogue between the assistant 1002 and the user that runs through other procedures, such as flow model dialogues and objectives. The choice of constraints provides the infrastructure for linking models of the flow of conversations and tasks at this level of abstraction (i.e., appropriate for the tasks of selection with restrictions regardless of the subject area).

[0392] Referring now to Fig.46, shown is an example of dialogue 4600, to help direct the user through the search process, so that the relevant input data can be obtained.

[0393] In an exemplary dialogue 4600 first stage consists in giving the user what he is looking for, i.e. the class of choice. For example, the user can perform this by means of expressing "dinner in Palo Alto". This allows the assistant 1002 infer 4601 task and the subject area.

[0394] After the assistant 1002 understand the binding between the task and the subject area (class choice=restaurants), the next step is to understand what selection criteria are important to that user, for example by requesting 4603 of criter�EV and/or restrictions. In the above example, "Palo Alto" points of interest location. In the context of restaurants, the system may interpret the location as a proximity restrictions (technically, restrictions on the proximity criterion). Assistant 1002 explains that it is necessary, accepts input. If there is enough information to limit the set of choices to a reasonable size, then assistant 1002 paraphrase enter 4605 and is one or more restaurants that satisfy the constraint location, sorted in a useful order. The user can then choose 4607 from this list or to clarify 4606 criteria and constraints. Assistant 1002 talks about the above mentioned limitations and uses depending on the subject area knowledge in order to recommend other criteria that can help, also request restrictions on these criteria. For example, assistant 1002 may reason that if the recommendations of the restaurants on the distance that you can walk from the hotel, useful criteria for a query must be the availability of table and kitchen.

[0395] the selection Problem with constraints is executed when the user selects 4607 instance of the selection class. In one embodiment of the implementation, additional further tasks 4602 provided by assistant 1002. Still� way assistant 1002 can offer services that indicate the choice in the provision of some other value. Examples 4608 is a restaurant reservation, set a reminder on your calendar and/or sharing with others through choice of sending the invitation. For example, the order clearly indicates that it is selected; other options may be to put a restaurant in the calendar or send the invitation with directions on how to get, friends.

[0396] Referring now to Fig.47 shows the block diagram of the sequence of operations illustrating a method of choice with constraints according to one embodiment of the. In one embodiment, the implementation, the assistant 1002 adaptable works with intermittent initiative, allowing the user to navigate to the internal circuit, e.g., including objectives, topics, criteria and constraints for one or more at a time in the input.

[0397] the Method begins 4701. The input is accepted from the user 4702 according to any one of the modes described in this document. If, on the basis of the input task is unknown, the assistant 1002 requests 4705 clarifying input from the user.

[0398] In step 4717 assistant 1002 determines, or provides no additional user input. If so, the assistant 1002 returns to step 4702. In �rotinom case, method ends 4799.

[0399] If at step 4703, the task is known, the assistant 1002 determines 4704 whether or not the task selection with restrictions. If not, the assistant 1002 passes 4706 to the specified task flow.

[0400] If at step 4704 task is the selection of constraints, assistant 1002 determines 4707 whether or not the selection class to be defined. If not, the assistant 1002 offers 4708 selection of well-known classes of choice and returns to step 4717.

[0401] If in step 4707 class selection may be determined, assistant 1002 determines 4709, may or not be defined all the mandatory restrictions. If not, the assistant 1002 requests in the form of tips 4710 requested information and returns to step 4717.

[0402] If at step 4709 all required constants can be determined, assistant 1002 determines 4711, or not to be found any elements of the results, subject to the restrictions. If there are no items that meet the constraints, assistant 1002 offers 4712 ways to ease them. For example, assistant 1002 may ease the restrictions from lowest to highest priority using the algorithm of the filtering/sorting. In one embodiment, the implementation, if there are elements that satisfy some constraints, assistant 1002 can rephrase the situation (conclusion, for example: "I can't find rivers�recommended Greek restaurants which deliver on Sundays in San Carlos. However, I found 3 Greek restaurant and 7 of the recommended restaurants in San Carlos".). In one embodiment, the implementation, if there are no items that match any of the constraints, assistant 1002 can rephrase the situation and request in the form of tips of different constraints (conclusion, for example: "Sorry, I can't find restaurants in Anytown, Texas. You can choose another location."). Assistant 1002 returns to step 4717.

[0403] If in step 4711 result items can be found, assistant 1002 offers 4713 list of items. In one embodiment, the implementation, the assistant 1002 paraphrase the current set criteria and limitations conclusion, for example, "Here are some recommended Italian restaurants in San Jose". (recommended=Yes, cuisine=Italian, proximity=<in San Jose>)). In one embodiment, the implementation, the assistant 1002 is sorted, paged list of items that satisfy the known constraints. If the item only shows some limitations, such condition can be shown as part of display items. In one embodiment, the implementation, the assistant 1002 offers the user a means to select an item, for example, by initiating another task for that item, such �AK order reminder, scheduling or sharing. In one embodiment of the implementation, for any given element, the assistant 1002 represents the characteristics of elements that are characteristic of a sample of instances of a class choice. In one embodiment, the implementation, the assistant 1002 shows how the element satisfies the constraint; for example, a rating in Zagat satisfy constraint 5 "recommended=Yes" and "at a distance of 1 mile" restriction "on the distance that you can walk from address". In one embodiment, the implementation, the assistant 1002 allows the user to drill down for more details for the item, resulting in display of a greater number of signs of elements.

[0404] the Assistant 1002 determines 4714, selects or not the user element. If the user selects an item, the task is completed. Any further task is performed 4715, if any, and the method ends 4799.

[0405] If in step 4714 the user selects an item, the assistant 1002 offers 4716 ways for users to choose other criteria and constraints, and returns to step 4717. For example, given the current set of criteria and constraints, assistant 1002 can offer criteria that are most likely to limit the set of choices required to�th dimension. If the user selects the limit value, the limit value is added to the previously determined limits, when stages 4703-4713 again.

[0406] Since the one or more criteria may have inherent value preferences, the selection criteria may add information to the query. For example, enabling the user to indicate that the positive reviews are appreciated, allows the assistant 1002 to sort by this criterion. This information can be taken into account when the stages 4703-4713 again.

[0407] In one embodiment, the implementation assistant 1002 enables the user to increase the importance of the criterion, which has already been specified, so it should have a higher priority order. For example, if the user requests fast, cheap, highly recommended restaurants within one block from your location, assistant 1002 may request that the user chose, which of these criteria is more important. This information can be taken into account when the stages 4703-4713 again.

[0408] In one embodiment, the implementation, the user may provide additional input at any moment at the time when the method according to Fig.47. In one embodiment, the implementation, the assistant 1002 checks periodically or continuously to the subject of such an a� input and, in response, goes back to step 4703 to process it.

[0409] In one embodiment of implementation, the derivation of the element or list of elements, assistant 1002 indicates in the representation of the elements of the signs, which are used to select and organize them. For example, if a user requests a nearby Italian restaurants, such indication elements for distance and kitchens can be shown in the item view. This may include highlighting and a list of selection criteria, which are involved in the representation of the element.

Approximate subject area

[0410] table 1 provides an example of the subject areas of choice with constraints that can be handled by assistant 1002 according to various embodiments of the implementation.

Filtering and sorting results

[0411] In one embodiment, the implementation in the presentation of the elements that meet the current criteria and constraints can be used technology filtering/sorting. In one embodiment, the selections may serve as parameters for filtering and sorting for basic services. Thus, any selection criterion may be used to determine which elements are in the list, � to calculate the order where you can split them into pages and show. The sort order for this task is akin to the relevance rank for search. For example, proximity is a criterion with character limits, for example, a "distance that can be driven on a car" and the General concept of sorting by distance. Limiting the distance that one can drive" can be used to select a group of elements. In this group of closely located elements can be sorted in the list above.

[0412] In one embodiment of the implementation, selections and associated filtering and sorting are discrete "levels", which are functions from the underlying data and the user input. For example, the proximity are grouped into levels, such as "the distance that you can walk", "the distance that can be reached by taxi", "the distance that can be driven on a car". When sorting one or more elements on the distance that you can walk, are treated as if they are at the same distance. The input from the user may come into play in the way he specifies the limit. If the user enters, for example, "Palo Alto", one or more elements within the city limits of Palo Alto are the perfect match and are�I equivalent. If the user enters "near the railway station on University Avenue, the match should depend on the distance from that address, wherein the degree of overlap depends on the class of choice (e.g. "about" for restaurants differs from the "about" for hotels). Even within the constraints that can be specified with a continuous value that can be used discreteness. This can be important for sort operations, so that multiple criteria can participate in the definition of the ordering on the criterion of best fit.

[0413] In one embodiment of implementation, the list of items (items that are considered to be "coincident" or "good enough") can be smaller or larger than the number of items displayed on one "page" of output. In General, elements on the first page is given the most attention, but conceptually there is a longer list, and the pagination is simply a function of form factor media output. This means, for example, that if the user is offered a way to sort or view items according to a certain criterion, then it is a complete set of elements (multiple pages) is sorted or viewed.

[0414] In one embodiment of implementation, criteria for selection, there is the ordering priority. In other words, some criteria may have �greater value than others in filtering and sorting. In one embodiment, the implementation, the criteria selected by the user, are granted a higher priority than others, and provides for the streamlining of default on one or more criteria. This provides the lexicographical sorting. The assumption is that there is a significant priori priority. For example, if the user specifies otherwise, for the restaurant may be more important to be closer than to be inexpensive. In one embodiment, the implementation, the ordering of the a priori preference depends on the subject area. The model provides the opportunity for specific user preferences to override the default values of the subject area, if required.

[0415] Since the values of constraints can represent multiple internal data types, there are different ways of matching restrictions, and they can be for specific restrictions. For example, in one embodiment, the implementation of:

- Binary constraints coincide as one or more or none. For example, it has a restaurant type of "fast food", may be true or not.

- Restrictions of membership in the set same as one or more or none based on the value of properties. For example, kitchen=Greek means a set of dishes for the restaurant� includes Greek.

- Restrictions enumeration coincide with a threshold value. For example, the criterion of a rating may have value restrictions "rated", "top rated" or "highest rated". The restriction as "highly rated" also matches "highest rated".

- Numeric limits coincide with a threshold value that may be of particular criteria. For example, "open late" can be a criterion, and the user can request places open after 22:00. This type of restriction may be a little beyond the task of selection with restrictions, because it's not a character limit. However, in one embodiment, the implementation, the assistant 1002 detects more cases of numeric limits, such as these, and converts the threshold value with character restrictions (for example, "restaurants in Palo Alto now open"->"here's 2 restaurant in Palo Alto which are open until late").

- The location and time is handled in a special way. The restriction on the proximity may be of interest to the location specified with some degree of detail, which determines the coincidence. If the user specifies a city, the match at the city level is appropriate; zip code can provide the radius. Assistant 1002 may also understand location�tion, that are "about" other interesting locations, also on the basis of special processing. Time is relevant as a constraint value criteria that have a threshold value on the basis of the service invocation, such as the availability of a table or the flights within a given time range.

[0416] In one embodiment of implementation, the constraints can be modeled so that there is one threshold value to select a small set of discrete values for sorting. For example, the criterion of affordability can be modeled as approximately binary limitations if affordable restaurants - all restaurants below some threshold range of prices. When data is backed up by several discrete levels for choice, constraints can be modeled using the gradient of the match. In one embodiment, the implementation may be provided two levels of compliance (for example, strict and non-strict matching); however, specialists in the art should recognize that other embodiments may be provided any number of levels of compliance. For example, proximity may be consistent with a fuzzy boundary, so that the essence, which are located near for the location can be the same as lax. Practical PEFC�subsequently the strict or fuzzy matching algorithm is to filtering/sorting, as described below.

[0417] At least one criterion for the approach to the match threshold and default values may be established, if relevant. The user may be able to just pronounce the name for the symbolic value of the limit or the exact constraint expression, if it is treated specially (for example, time and location).

[0418] the Ideal situation for choice with limitations occurs when the user specifies the restrictions that lead to a short list of options, one or more of which satisfies the constraints. The user then selects the winners based on the signs of the elements. In many cases, however, the problem is excessive or does not have sufficient limits. When it has excessive restrictions, there is little or no elements that satisfy the constraints. When she does not have sufficient limits, there are so many options that the analysis list is not appropriate. In one embodiment of implementation, the General model of choice with limitations of the present invention has the ability to handle multiple constraints with reliable matching and usually forming something from which one can choose. Then, the user may prefer to specify their criteria and constraints or simple� to perform the task with "good enough" solution.

Method

[0419] In one embodiment of the implementation, a method is used to filter and sort the results:

1. Considering an ordered list of selection criteria chosen by the user, determining the limitations of at least one.

a. If the user has specified a value for the limit, use it. For example, if the user uttered "Greek food", the limit is the kitchen=Greek. If the user said "San Francisco", the restriction is "In the city of San Francisco." If the user said "South of market", then the limit is In the vicinity of SoMa.

b. Otherwise, use-dependent subject field and criteria default values. For example, if the user uttered "table in some Thai school", indicating that the criterion of accessibility is relevant, but he did not specify a limit value. Values default constraints for accessibility can be some range of times and dates, for example, tonight, and the number of participants soiree default 2.

2. The choice of the minimum of N results through such limitations.

a. Attempting to get N results in strict compliance.

b. If it is unsuccessful, an attempt to ease the restrictions in the reverse order of priority. Ins�and words, coincidence on a strict level for one or more of the criteria except the last, which may be the same as at ordinary level. If no soft match for this limitation, we attempted to use non-strict matches up line from lowest to highest priority.

c. Then the repetition of the circuit, allowing the error in the coincidence limit, from lowest to highest priority.

3. After obtaining the minimum set of choices of a lexicographic sort on one or more criteria (which may include the user-specified criteria, and other criteria) in order of priority.

a. Consideration to set the user-specified criteria as highest priority, then one or more of the remaining criteria in their a priori priority. For example, if a priori priority is (accessibility, kitchen, proximity, rating), and the user provides constraints on the proximity and the kitchen, the sort priority is (kitchen, proximity, availability, rating).

b. Sorting criteria for using discrete levels of compliance (strict, lax, no) using the approach, the identical approach to easing of restrictions, this time using the complete list of criteria.

i. If the set of choices is obtained without the donkey�ing restrictions one or more of the set of choices can "connect" to sort, because they represent one or more matches on strict levels. Further, the following criteria in the priority list may become effective to sort them. For example, if the user says that the cuisine = Italian, proximity = In San Francisco, and the sort priority is (kitchen, proximity, availability, rating), one or more places in the list have equal values match for kitchen and proximity. Thus, the list should be sorted by distance (space available with tables appear first). Inside of places available, the places with the highest rating should be at the top.

ii. If the set of choices obtained through the relaxation of restrictions, then one or more identical elements placed at the top of the list, followed by overlapping elements. In matching the group they are sorted through the remaining criteria, as in the overlapping group. For example, if there are only two Italian restaurant in San Francisco, there are shown first and then unavailable. Then show the rest of the restaurants in San Francisco, sorted by distance and rating.

Organize by priority

[0420] the Technology described in this document provide�will be responsible for swarnajayanti assistant 1002, despite the indicated limitations and incomplete data. In one embodiment, the implementation, the assistant 1002 uses these technologies to create a custom list of items in order according to the criterion of best fit, i.e. according to relevance.

[0421] In one embodiment of the implementation, such a sorting by relevance based on a priori regularization priority. In other words, entities that are important regarding the subject area, a set of criteria is selected and placed in order of importance. If one or more entities that match the criteria above in order of priority may be more relevant for the choice with the restrictions of the elements, the lower in order. Assistant 1002 may operate based on any number of criteria. In addition, the criteria can be modified in time without violating existing modes of behavior.

[0422] In one embodiment, the implementation, the order of precedence for the criteria can be adjusted depending on the subject area parameters, because the manner in which interact the criteria may depend on the class of choice. For example, when choosing hotels, availability and price can be the dominant constraints, whereas in the case of restaurants, the cuisine and the proximity may be more important.

[0423] In one embodiment, implementation, user� can override the ordering criteria by default in the dialog. This enables the system to direct the user, when the search has excessive restrictions, through the use of sequencing, to determine what restrictions should be attenuated. For example, if the user specifies constraints on the kitchen, proximity, recommendation and dish, and no fully matching entries, the user can tell that the dish is more important than the level of recommendation, and change the combination so that the desired compliance of the dishes are sorted first.

[0424] In one embodiment of the implementation, when the priority order is determined, the user-specified constraints have priority compared to other. For example, in one embodiment, the implementation, the proximity is a binding constraint and thus always indicated and additionally has priority over other non-selected constraints. Therefore, do not necessarily have to be a limitation with the highest priority for the strong dominance. In addition, many criteria may not be the same as one or more, if the restriction is not set by the user, and thus, the priority of these criteria has meaning only in user-selectable criteria. For example, when the user specifies the kitchen, it is important for them, but otherwise irrelevant to sort the elements.

[0425] for Example, the following is Paradis�mA sort by variants of priority for the subject area restaurants:

1. kitchen* (not sortable, if the limit value is not set)

2. availability* (sortable using the values of the default restrictions, for example, time)

3. recommended

4. proximity* (the limit value is always set),

5. affordability

6. the ability to deliver

7. the dish (not sortable, if the limit value is, for example, the keyword is not set)

8. keywords (not sortable, if the limit value is, for example, the keyword is not set)

9. the name of the restaurant

[0426] the Following is an example of diagrams for explaining the above paradigm of the sort:

- If the user specifies the kitchen, he wants to remember it.

- If one or more things are the same, sort by rating (highest priority for the criteria can be used for sorting without limitation).

- At least one variant of implementation, the proximity may be more important than most entities. However, because it coincides at discrete levels (in the city, within a radius for passage on foot, etc.), and it is always specified, then most of the time, most of the matching elements can be "attached" in the vicinity.

- Accessibility (defined by the search web site, �ACOM as, for example, open-table.com) is a valuable sorting criteria may be based on the default value for sorting, if not specified. If the user specifies the time to order, only available places can be on the list, and sorting can be based on the recommendations.

- If the user says he needs the most recommended places, he can sort the above proximity and accessibility, and these criteria may diminish before recommendation. The assumption is that if someone is looking for a good place, he may be willing to go a little further, and this is more important than the availability of a table by default. If a specific time for availability is specified, and the user requests recommended places, places that are recommended and available, you can go first, and the recommendation may be attenuated at lax compliance before availability becomes the same as one or more.

- The remaining restrictions except the names are one or more on the basis of incomplete data or overlap. So they represent a heuristic lax default sort, and when they are specified - match one or more or none".

- The name can be used as constraints to handle the case in which apomine�Xia restaurant by name, for example, one or more of Hobee's restaurants near Palo Alto. In this case, one or more elements to have the same name and can be sorted by proximity (another specified limitation in this example).

The domain modeling: convert selection criteria in the underlying data

[0427] it May be desirable to distinguish between data that are available for the calculation of assistant 1002, and data used to implement the choice. In one embodiment, the implementation, the assistant 1002 uses a data model that reduces the complexity for the user by coagulation of one or more types of data used to distinguish the elements in a simple model selection criteria. Internally, this data can take several forms. Instances of the class can select to have internal properties and attributes (for example, the kitchen of the restaurant), can be compared along the dimensions (for example, the distance from a certain location) and can be detected by means a request (for example, they coincide or not with the design or text is available or not at this time). They can also be computed from other data that is not presented to the user as selection criteria (e.g., weighted combinations of ratings from multiple sources�of ICA). These data are relevant as "one or more" for the task, but the differences between these three types of data are not relevant for the user. Because the user thinks from the point of view of characteristics of the desired choices, and not the properties and dimensions, the assistant 1002 operationaliseren these different criteria in the signs of the elements. Assistant 1002 provides a user-friendly data model subject area and converts it to data found in web services.

[0428] One type of transformation is an isomorphism of the underlying data in a user-friendly criteria. For example, availability of tables for backup, observed by the user can be exactly the same as offers web site, online reservation, for example, opentable.com using identical granularity for time and number of participants soiree.

[0429] Another type of transformation is to normalize the data from one or more of the services in the set of common values, possibly with the unification of equivalent values. For example, the cuisine of one or more restaurants can be submitted as a single ontology in the assistant 1002 and converted to various dictionaries used in the various services. This ontology can be hierarchical and have end caps, pointing to a specific �values, at least one service. For example, one service may have a value of cuisines - "Chinese" and "Szechuan" and the third is "Asian". The ontology used by assistant 1002, and shall cause the reference to "Chinese food" or "Szechuan" semantically coincide with one or more of these nodes with a confidence levels reflecting the degree of coincidence.

[0430] the Normalization may also be involved in the elimination of differences in precision. For example, the location of the restaurant may be provided at street level in one service, but only as a city in the other. In one embodiment, the implementation, the assistant 1002 uses deep structural representation of locations and times, which can be converted to various surface data values.

[0431] In one embodiment of implementation, the assistant 1002 uses a special kind of conversion for open qualifiers (e.g., romantic, quiet), which can be converted to match full-text search, tags, or other open-textured signs. The name for the selection in this case should be something like "describes".

[0432] in at least one embodiment of the implementation, constraints may be converted to organise practical preferences. In other words, given the trade name� selection criteria and limit values, assistant 1002 has the ability to interpret the criterion as the ordering on the possible elements. There are several technical issues that should be resolved at such transformation. For example:

- Ordering by preference may conflict. The ordering provided by one constraint may be inconsistent or even inversely correlated with the ordering provided by the other. For example, price and quality tend to be the opposite. In one embodiment, the implementation, the assistant 1002 interprets the constraints selected by the user in a suspended, or otherwise combined to streamline that reflects the desires of the user, but is true for the data. For example, a user may request "cheap French fast food restaurants on the distance that you can walk, with a high rating. In many locations there may not be such a restaurant. However, in one embodiment, the implementation, the assistant 1002 may display a list of items, which strives to optimize, at least one constraint, and explain why at least one element is transferred. For example, one element may be "French cuisine with a high rating" and the other "cheap cafe bistro� power at a distance, which you can walk".

- Data can be used either as hard or soft constraints. For example, the price range of the restaurant could be important for selection, but may be difficult to specify a threshold for the top prices. Even such seemingly harsh restrictions, such as kitchen, in practice, can be soft constraints due to partial matches. Because in one embodiment, the implementation assistant 1002 uses a data modeling strategy that seeks to mitigate one or more of the criteria in character values (for example, "cheap" or "close"), these constraints can be transformed into a function that correctly interprets the criteria and procedure without rigor in coincidence with specific threshold values. For character criteria with a clear objective values of the truth of the assistant 1002 can assign objective criteria more weight than other criteria, and to clarify in the explanation that he was aware that some elements do not coincide strictly with the requested criteria.

Elements can coincide with some, but not with one or more restrictions, and can be shown the elements of a "best approximation".

- In General, the assistant 1002 determines which traits are characteristic of �redmetal region and which can serve as selection criteria and, at least for one criterion, the possible values of the constraints. This information may be provided, for example, using operational data and API calls.

The text of perefrazirovanie and strings hint

[0433] As described above, in one embodiment, the implementation assistant 1002 provides feedback to show that he understands the intention of the user and works towards the goals of the user by generating perefrazirovanie his current understanding. In models of spoken dialogue of the present invention paraphrasing is that assistant 1002 displays after the user input the Preface (for example, paraphrasing 4003 in Fig.40) or summaries of results that should be considered (e.g. list 3502 in Fig.35). A line with the tip is the recommendation to the user about what he can do to clarify your request or to explore the space of choice along some dimensions.

[0434] In one embodiment, the implementation, the purpose of the text perefrazirovanie and rows with tip include, for example:

- to show that assistant 1002 understands the concepts in the user input, and not just text;

- specify the boundaries of understanding assistant 1002;

- direct the user to enter the text that you want for the intended task;

- to help the user learn�ü the space of possibilities when choosing to restrictions;

- to explain the current results obtained from the services, from the point of view of the established criteria of the user and admissions assistant 1002 (for example, to explain the results of the queries without sufficient or excessive restrictions).

[0435] for Example, the following paraphrase and a line with a clue illustrate a few of these goals:

User input:

indonesian food in menlo park (Indonesian food in Menlo Park)

Interpretation system:

Task=constrainedSelection

SelectionClass=restaurant

Restrictions:

Location=Menlo Park, CA

Cuisine=Indonesian (known in the ontology)

The results of the services:

there is no strict correspondence

Paraphrasing:

Sorry, I can't find any Indonesian restaurants near Menlo Park (Sorry, I can't find any Indonesian restaurants near Menlo Park).

A string with the prompt:

You can try other cuisines or locations (You can try other cuisine or location).

The string with the tip under the hypertext links:

Indonesian (Indonesian): You can try other food categories such as Chinese or the favorite food item such as steak (You can try other types of food, for example, Chinese, or favorite dish, for example, steak).

Menlo Park (Menlo Park): Enter the location such as the city, neighborhood, street address or "near" followed by the landmark (Enter a location, for example, the city, the neighborhood, the actual address or "about", then the attractions).

Cuisines (Food): Enter the food category such's Chinese or Pizza (Enter a category of food, for example, Chinese or pizza).

Locations): Enter the location: the city, zip code or "near" followed by the name of the place (Enter a location: a city, zip code or "about", then the name of the place).

[0436] In one embodiment of implementation, the assistant 1002 responds to user input relatively quickly with paraphrasing. Paraphrasing is then updated once the results are known. For example, the initial response may be "Search Indonesian restaurants near Menlo Park...". Once results are obtained, the assistant 1002 should update the text to read "Sorry, I can't find any Indonesian restaurants near Menlo Park. You can try other cuisines or locations. It should be noted that certain elements are highlighted (indicated here by underlining), indicating that these elements represent limitations that may diminish or alter.

[0437] In one embodiment, the implementation of special formatting/highlighting is used for key words in the paraphrase. This can be useful to simplify the learning user to interact with intelligent automated assistant 1002 by specifying to the user which words are most important and are more likely to be recognized by assistant 1002. User�then spruce may be more likely to use such words in the future.

[0438] In one embodiment, the implementation of paraphrasing, and the string with the tip formed using any relevant contextual data. For example, any of the following data elements may be used, alone or in combination:

- Grammatical analysis - the tree node of the ontology that is bound to a matching input tokens, notes and exclusions. For each node in the grammatical analysis, this may include the metadata of the node and/or all tokens in the input, which provide a rationale for the value of a node.

- The problem, if known.

Class choice.

- The restriction of location, regardless of class choice.

- What are the mandatory parameters are unknown for a given selection class (e.g., a mandatory limit on the restaurants).

- The name of the object name in the grammatical analysis, which is an instance of the selection class, if still exist (e.g. the name of a specific restaurant or movie).

Is further clarification or beginning of the conversation? (Reset starts a new conversation).

- What are the limitations for grammatical analysis are bound to the values in the input that changed their values? In other words, what restrictions simply changed through the last entry?

Class of choice is logically deduced �whether directly specified?

- Sorted for quality, relevance or proximity?

- For each specified limits of how well it matches?

- Clarification is entered as text or click?

[0439] In one embodiment, the implementation, the algorithm consider rephrasing the query, model 1056 subject area and the results of providing services. The model 1056 subject area contains classes and attributes, including metadata that is used to determine how to shape the text. Examples of such metadata for the formation of paraphrasing include:

- IsConstraint={true | false}

- IsMultiValued={true | false}

- ConstraintType={EntityName, Location, Time, Category Constraint, AvailabilityConstraint, BinaryConstraint, SearchQualifier, GuessedQualifier}

- DisplayName=string

- DisplayTemplateSingular=string

- DisplayTemplatePlural=string

- GrammaticalRole={AdjectiveBeforeNoun,Noun,ThatClauseModifer}

[0440] for Example, grammatical analysis may contain these elements:

Class: Restaurant

IsConstraint=false

DisplayTemplateSingular="restaurant"

DisplayTemplatePlural="restaurants"

GrammaticalRole=Noun

Feature: RestaurantName (example: "Il Fornaio")

IsConstraint=true

IsMultiValued=false

ConstraintType=EntityName

DisplayTemplateSingular="named $1"

DisplayTemplatePlural="named $1"

GrammaticalRole=Noun

Feature: RestaurantCuisine (example: "Chinese")

IsConstraint=true

IsMultiValued=false

ConstraintType=CategoryConstraint

GrammaticalRole=AdjectiveBeforeNoun

Feature: RestaurantSubtype (example: "cafe")

IsConstraint=true

IsMultiValued=false

ConstraintType=CategoryConstraint

DsplayTemplateSingular="$1"

DisplayTemplatePlural="$1s"

GrammaticalRole=Noun

Feature: RestaurantQualifiers (example: "romantic")

IsConstraint=true

IsMultiValued=true

ConstraintType=SearchQualifier

DisplayTemplateSingular="is described as $1"

DisplayTemplatePlural="are described as $1"

DisplayTemplateCompact="matching $1"

GrammaticalRole=Noun

Feature: FoodType (example: "burritos")

IsConstraint=true

IsMultiValued=false

ConstraintType=SearchQualifier

DisplayTemplateSingular="serves $1"

DisplayTemplatePlural="serve $1"

DisplayTemplateCompact="serving $1"

GrammaticalRole=ThatClauseModifer

Feature: Recommended (example: true)

IsConstraint=true

IsMultiValued=false

ConstraintType=BinaryConstraint

DisplayTemplateSingular="recommended"

DisplayTemplatePlural="recommended"

GrammaticalRole=AdjectiveBeforeNoun

Feature: RestaurantGuessedQualifiers (example: "spectacular")

IsConstraint=true

IsMultiValued=false

ConstraintType=GuessedQualifier

DisplayTemplateSingular="matches $1 in reviews"

DisplayTemplatePlural="match $1 in reviews"

DisplayTemplateCompact="matching $1"

GrammaticalRole=ThatClauseModifer

[0441] In one embodiment of implementation, the assistant 1002 is able to process non-matching input. In order to process the input, the model 1056 subject area may provide the node type GuessedQualifier for each class of choice and rules that are the same in other respects, mismatched words, if they are in correct grammatical context. In other words, GuessedQualifiers are treated as different nodes in the grammatical analysis, which coincide when there are words that are not found in the ontology, but which are in great�ilen context, to indicate that they are likely to be class specifiers choice. The difference between GuessedQualifiers and SearchQualifiers is that the latter are mapped to the vocabulary of the ontology. This difference gives the opportunity to paraphrase what the assistant 1002 clearly identified intent SearchQualifiers, and may be less sustainable in the reflection GuessedQualifiers.

[0442] In one embodiment of implementation, the assistant 1002 performs the following steps during the formation of the text rephrasing:

1. If the task is unknown, an explanation of what the assistant 1002 can do, and the query in the form of prompts for additional input.

2. If task is the selection problem with constraints, and the location is known, then the explanation of the subject areas that the assistant knows 1002, and the query in the form of tips for the class of choice.

3. If the grade of choice is known, but the binding constraint is missing, the query in the form of hints for this constraint (for example, location is required for choice with restrictions.

4. If the input contains EntityName class selection, then the output of "search"<name><location>.

5. If this is the initial query in the conversation, then the conclusion is "search", then complex nominal groups, which describes the restrictions.

6. If this is a stage further clarification in the dialog:

a. If the user then�Ko finished the required input then the conclusion "thank you" and then paraphrasing. (This occurs when there is a binding constraint, which is converted to user input).

b. If the user changes the limit, the proof of this and then paraphrasing.

c. If the user has entered the proper name of the class instance selection, processing it in a special way.

d. If the user simply adds the unrecognized phrase, specifying how it should be dismantled as search. Optionally, the input may be coordinated to the search service.

e. If the user simply adds the normal limit, the output is "Good" and normal paraphrasing.

7. To explain the results, using an identical approach to paraphrasing. However, when the results are surprising or unexpected, the explanation of the results using knowledge about the data or the service. In addition, when the request is excessive or does not have sufficient limits, the query in the form of prompts for additional input.

Grammar to form a complex noun groups

[0443] In one embodiment of the implementation, when paraphrasing 734 request in a selection problem with constraints is based on a complex nominal group around the class selection, which refers to the current restrictions. Each�th constraint has grammatical position on the basis of its type. For example, in one embodiment, the implementation, the assistant 1002 may be paraphrasing, for example:

recommended romantic Italian restaurants near Menlo Park with free tables for two that offer the Osso Buco and are described as "quiet"

Grammar to compile it, the following:

<paraphraseNounClause>:==<binaryConstraint><searchQualifier><categoryConstraint><itemNoun><locationConstraint><availabiltyCon-straint><adjectivalClauses>

<binaryConstraint>:==one adjective that indicates the presence or absence of BinaryConstraint (for example, the recommended (best), affordable (cheap)) you Can list a few in a single request.

<searchQualifier>:==the word or words that match the ontology for the class specifier choice, which must be passed to the service search engine. (e.g., romantic restaurants, funny movies). Use when ConstraintType=SearchQualifier.

<categoryConstraint:==an adjective that identifies the genre, the kitchen or the category class of your choice (for example, a Chinese restaurant or a file with the R rating). This is the last adjective prefix, because it is the most significant. Use a for characteristics of the type CategoryConstraint and GrammaticalRole=AdjectiveBeforeNoun.

<itemNoun>:==<namedEntityPhrase>|<selectionClass>| <selectionClassSubType>. Find the most specific method to display a noun.

NamedEntity< SubType<Class

<selectionClass>:==noun, which is a generalized name for a class of choice (e.g. restaurant, movie, place)

<selectionClassSubType>:==registered group which is a subtype of class choice, if known (e.g., diner, Museum, store, bar for class select "local organization") to Use for signs of where ConstraintType=CategoryConstraint and GrammaticalRole=AdjectiveBeforeNoun.

<namedEntityPhrase>:==<fieldName> | "the" (<selectionClass> | <selectionClassSubType>)

<fieldName>:==the proper name of the instance of the selection class (e.g., "Il Fornaio" "Animal House" "Harry's Bar")

<locationConstraint>:==<locationPreposition><locationName>

<locationPreposition>:=="in", "about", "on", etc.

<locationName>:==city, an address, a landmark, or something for GPS, for example, "Your current location"

<availabilityConstraint>:==limiting distance, expressed as a prepositional phrase that comes after the noun (e.g., "free tables", "available locations", "available online"). It comes immediately after a noun to give it the characteristic properties.

<adjectivalClauses>:==<modiferVerbPhrase> | "that" <modiferVerbPhrase> "and" <modiferVerbPhrase>

<modiferVerbPhrase>:=verbal group, which expresses a limitation in the style of keywords to search for the selection class (e.g., restaurants that "are described as a quiet", "offer meats� after 11" coincident with 'incredibly fashionable' in review'; movies that contain violence", "starring Billy Bob Thornton"). Enable all constraints in grammatical analysis, for which GrammaticalRole=ThatClauseModifer, by using a variation of "that...and", if more than one. Use DisplayTemplatePlural to form of expression "that", putting GuessedQualifier last. If there is only one such constraint, use a variant DisplayTemplateCompact.

[0444] table 2 provides several examples of perefrazirovanie that may be provided in response to the first input for the task, according to one variant of implementation.

Clarification request about places to eat

[0445] table 3 provides some examples of perefrazirovanie in response to the situation, when the user intends to find a place to eat, you know, but he has not selected a specific place to eat. He displayed a list of restaurants, and it provides clarification.

Table 3
Rephrasing in response to clarification
Refine
Menlo Park (was unknown)Thank you, now search restaurants in IU�lo Park.
cheapestWell, the search for affordable restaurants near Menlo Park.
Chinese (was Italian)Well, changing the kitchen in Chinese. Search Chinese restaurants near Menlo Park.
pizza (adding food item)Well, a search for Italian restaurants near Menlo Park that offer pizza.
San Francisco (Menlo Park)Well, change the location to San Francisco. Search Italian restaurants near San Francisco that offer pizza.
terribly fashionableWell, I'll have a look at the reviews "terribly fashionable".
quiet romanticWell, the search for romantic restaurants in Palo Alto described as "quiet"
cliff houseCliff House looks familiar. Look for him in Palo Alto.

[0446] table 4 provides some examples of summaries of results are provided as soon as results are obtained.

[0447] table 5 provides some examples of straw� with tip, granted, when users click the clickable links.

Line with a prompt when users click active links

Recommendation of possible answers in the dialogue

[0448] In one embodiment, the implementation assistant 1002 provides contextual recommendations. Recommendations are a way for the assistant 1002 to offer the user the options to move from their current situation in the dialogue. A set of recommendations, proposed by assistant 1002, depends on the context and the number of the proposed recommendations may depend on the carrier and form factor. For example, in one embodiment of implementation, the majority of distinctive recommendations could be offered in accordance with generally accepted practice in dialogue, an extended list of recommendations ("more") may be offered in the menu is scrollable, and additional recommendations can be done by typing a few characters and select from options for the autocomplete. Specialists in the art should recognize that other mechanisms may be used to provide recommendations.

[0449] In various embodiments, may be provided to different types of recommendations. Examples of the types of recommendations include:

- options to Refine the query, include�ie in the add or remove, or change the values of restrictions;

- options to fix or recover from incorrect situations, for example, "not what I meant" or "start again" or "search the Internet";

- options to resolve the ambiguity between:

- interpretations of speech;

- interpretations of the text, which includes the correction of spelling and semantic ambiguity;

- context-sensitive commands, for example, "show on map" or "send directions on how to get to my date" or "to explain these results";

- recommended recommendations for cross-selling, such as following multi-step scenarios of the organization of the dinner meals or events;

- options to reuse previous commands or parts thereof.

[0450] In various embodiments, the context that determines the most relevant recommendations that can be removed, for example, of the following:

• the state of the dialog

• the status of a user, including, for example:

static properties (name, home address, etc.)

- dynamic properties (location, time, network speed)

• background to the interaction, including, for example:

- the backstory of the query

history results

- the text that is typed before in the petrol�olnine.

[0451] In various embodiments, recommendations may be generated by any mechanism, such as, for example:

- paraphrasing the subject area, objectives or restrictions on the basis of the ontology model;

the clue's in the auto - completion based on the current subject area and restrictions;

- paraphrasing ambiguous alternative interpretations;

- alternative interpretations of converting speech to text;

- annotate manually on the basis of special conditions of dialogue.

[0452] According to one variant of implementation, the recommendations are generated as operations for teams in certain condition fill. Commands are explicit canonical representations of queries, including the assumptions and inferences made on the basis of interpretations for user input. In cases where user input is incomplete or ambiguous, the recommendations are an attempt to help the user to adjust the input to clarify the command.

[0453] In one embodiment, the implementation of each command is an imperative sentence with some combination of the following:

- command verb (imperative, for example, "find" or "where");

- subject area (class of choice, for example, "restaurants");

- limitation(I), for example, location�proposal = Palo Alto and cuisine = Italian.

[0454] These parts of the command (verb, subject area, constraints) correspond to nodes in the ontology.

[0455] the recommendation can then be considered as operations for the team, such as her job, change her or declaring that it is relevant or irrelevant. Examples include:

- job command-verb or subject area ("find restaurants"),

- change the command verb ("order it", show it on a map, save it"),

- change domain ("searching for a restaurant, not a local organization")

- an indication that the restriction is relevant ("you can Refine by cuisine"),

- select to limit ("Italian", "French", etc.)

- selection constraints and values together ("not far away", "tables for two")

- an indication of the limit value is incorrect ("this is not Boston")

- an indication that the restriction is not relevant (ignore expenses"),

- specifying the intention to change the limit value ("try another location"),

- change the constraint value ("Italian, not Chinese")

- Addendum to the limit value ("pool")

- bind the values to the grid ("Los Angeles, not Los Angelos")

- initiating a new command, repeated use�the stripes of the context ([after movie] "find nearby restaurants", "send directions how to get there, my friend"),

- initiation of a team that is "meta" context"to explain these results"),

- initiating a new command, reset or ignore context ("start again", "help with speech"),

[0456] also can contain some combination of the above. For example:

- "film Milk, and not [the restaurants] dish milk"

- restaurants offering pizza, not just pizza eatery"

- "A place called Costco in mountain view. I don't care whether You think it or not, that is a restaurant or a local organization"

- "the Chinese in mountain view" [recent request]

[0457] In one embodiment of implementation, the assistant 1002 includes a General mechanism to maintain the list of recommendations, ordered by relevance. The format in which it is proposed recommendation may differ depending on the current context, mode and form factor of the device.

[0458] In one embodiment of implementation, the assistant 1002 determines which restrictions to modify, by considering any or all of the following factors:

Is considering or has no value limitation;

- Considering the constraint is logically inferred or explicitly specified;

- Consideration of its characteristic properties (suggestionIndex)./p>

[0459] In one embodiment of implementation, the assistant 1002 determines the output format for recommendations. Examples of output formats include:

- change subject area:

- if the autocomplete option "find restaurants", "try something different"

- otherwise [logically deduced] "this is not a search of restaurants"

- change constraints in the form of names:

- if the logical name is displayed, the proposal is ambiguous alternative interpretations

- substitution in autocomplete of names of objects from the current results

- another name

- provided that this does not search by name (removal of restrictions) - you may offer a category instead

- "untitled"

"not in Berkeley"

- "some other day"

- not in that way (mixed use alternatives)

is logically deduced the date: any day, reservation is not needed"

[0460] In one embodiment of implementation, the assistant 1002 attempts to resolve the ambiguity through recommendations. For example, if the set of current interpretations of the user's intention is too ambiguous 310, the recommendations are one way to seek in the form of hints to obtain additional information 322. In one embodiment, the implementation for the tasks of selection constraints, assistant 1002 factorizes total limit for n�unambiguous interpretations of the intentions of 290 and represents the difference between the user. For example, if the user input includes the word "cafe", and this word may coincide with the name of the restaurant or type of restaurant, assistant 102 may ask: "You mean restaurants with the name 'cafe' or coffee shop-restaurants?"

[0461] In one embodiment of implementation, the assistant 1002 infers constraints in certain situations. In other words, for problems of a choice with restrictions, not all constraints need to be explicitly referred to in the user input; some may logically deduced from other information available in active ontology 1050, short-term storage device 1052 and/or other sources of information available to the assistant 1002. For example:

- The logical conclusion of the subject area or location

- The default assumption, such as the location of the

- Non-strict matching restriction (property location with a fuzzy and implicit characteristic properties, etc.)

- Ambiguous criteria (comparison with the limit value without a prefix (the name is compared with the category, often ambiguously))

[0462] In cases where the assistant 1002 infers the values of the constraints, it may also suggest that these assumptions as a recommendation for rejection by the user. For example, it can tell the user "I thought you meant the around. Want to look at another location?"

[0463] the Present invention is described with specific detail on possible options for implementation. Specialists in the art should take into account that the invention can be implemented in other embodiments. First, assign specific names to the components, capitalization of terms, the attributes, data structures, or any other aspect of programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Additionally, the system can be implemented through a combination of hardware and software, as described, or entirely in hardware elements, or completely in software elements. In addition, the particular division of functionality between the various system components described herein is merely indicative and not mandatory; functions performed by a single system component may instead performed by multiple components, and functions performed by multiple components may instead performed by a single component.

[0464] � various embodiments, the present invention can be implemented as a system or method for performing the above technologies, separately or in any combination. In another embodiment, the implementation, the present invention can be implemented as a computer program product containing a non-volatile machine-readable medium storing data and computer program code, encoded on the medium for instructing the processor in the computing device or other electronic device to perform the above described technology.

[0465] the Reference in this detailed description to "one implementation" or "an implementation option" means that a particular feature, structure or characteristic described in connection with the variant of implementation, included at least in one variant embodiment of the invention. Occurrences of the phrases "in one embodiment, the implementation" in various places of the detailed description do not necessarily refer to identical variant implementation.

[0466] Some portions of the above are presented from the point of view of algorithms and symbolic representations of operations of data bits in a storage device of the computing device. These algorithmic descriptions and representations are the means used by experts in the field of data processing for the most effe�tive to convey the essence of their work to others skilled in the art. The algorithm presented here and, in General, conceived as a self-consistent sequence of steps (instructions) leading to a desired result. Stages is something that requires physical handling of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals that allow the storage, transfer, combination, comparison and other processing. It is sometimes convenient, principally for reasons of normal usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. in addition, it is also convenient from time to time to mention specific layout stages that require physical treatments physical quantities as modules or encoders, without loss of generality.

[0467] However, you should take into account that all of these and similar terms should be associative associated with the appropriate physical quantities and are merely convenient labels that can be applied to these values. Unless explicitly stated otherwise, as apparent from the subsequent explanations should be taken into consideration that throughout the description, explanation, using terms such as "processing" or "calculating" or "displaying" or "determining" etc., indicate the action and processes of a computer system or similar electronic �facilitating module and/or device which processes and transforms data represented as physical (electronic) quantities in storage devices of a computer system or registers or other such storage, transmission or display information.

[0468] Specific aspects of the present invention include the stages and steps of the process described in this document in the form of an algorithm. It should be noted that the stages and steps of the process of the present invention may be implemented in software, firmware or hardware and the software can be loaded so that they are permanently housed and operated from different platforms used by a variety of operating systems.

[0469] the Present invention also relates to a hardware system to perform operations herein. This hardware system can be specially created for the required purposes or it may include computing a General purpose device selectively activated or reconfigured by a computer program stored in the computing device. Such a computer program may be stored on a computer-readable data storage media, such as, but not limited to, any type of disk including �you floppy disks, optical discs, CD-ROMs, magneto-optical disks, read-only memories (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, specialized integrated circuits (ASIC), or any type of media suitable for the preservation of electronic instructions, each of which is connected to a computer system bus. Additionally, as referred to in this document, the computing device may include a single processor or may be architectures that use a multiprocessor scheme for the expansion of computational capabilities.

[0470] the Algorithms and displays presented in this document is, in fact, not associated with any specific computing device, the virtualized system or other hardware system. Various system utility can also be used with programs in accordance with the ideas in this document, or may be convenient to construct more specialized hardware system to implement the required steps of the method. The required structure for many of these systems should be obvious from the description provided in this document. In addition, the present invention is not described in any specific programming language. Should take into account that many languages �programmirovaniya can be used for to sell the idea of the present invention, as described herein, and all of the above references to specific languages are provided for disclosure and best mode of the present invention.

[0471] Accordingly, in various embodiments, the present invention can be implemented as software, hardware, and/or other elements for controlling a computer system, computing device or other electronic device or any combination or multiple of the designated. Such an electronic device may include, for example, a processor, an input device (e.g. keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination of the above), an output device (e.g., screen, speaker, etc.), storage device, device long-term data storage (for example, a magnetic storage device, optical storage device, etc.) and/or network connection, according to the technologies that are known in the art. Such electronic device may be portable or deportations. Examples of electronic devices that can be used to implement the invention include: mobile phone, personal digital�first device, smartphone, kiosk, desktop computer, laptop computer, tablet computer, consumer electronic device, consumer electronic device; a music player; camera; TV receiver; STB; electronic gaming unit; etc. the Electronic device of the present invention may use any operating system, such as, for example, iOS or MacOS, Apple Inc., Cupertino, California, or any other operating system, which is made with the possibility of using the device.

[0472] Although the invention is described relative to a limited number of embodiments, specialists in the art using the advantages of the above description should take into account that can be developed other variants of implementation, which do not depart from the scope of the present invention, as described herein. In addition, it should be noted that the language used in the detailed description, mainly selected for readability and study and may be chosen to describe or limit the scope of the invention. Accordingly, the disclosure of the present invention has the intent to be illustrative and not limiting the scope of the invention as defined in the formula izobreteny�.

1. The method of operation of the intelligent automated assistant, comprising stages on which the electronic device comprising a processor and a memory that stores instructions for execution by the processor:
accept a user query, wherein the user request includes at least a speech input received from the user;
handle user request to retrieve the representation of the user's intention, and this view of the intention of the user connects the user's request with a task flow implementing the requested task, wherein the task flow is adapted to activate a variety of services, each of which supports the functions according to the corresponding set of parameters of the services;
identify, based on the submission of the user's intention, many relevant parameters of the problem for the flow of tasks, with many relevant parameters of the problem includes at least one task parameter, pronounced in speech input, and at least one task parameter, logically derived from the context information associated with speech input; and
perform the task flow by selective activation of a subset of a plurality of services in accordance with many relevant parameters of the problem, execution� task flow contains a stage, where, after unsuccessful attempts to perform the task in accordance with all of the many relevant parameters of the problem, choose one or more task parameters from the set of relevant parameters of the problem to mitigate with additional try to perform the task, and this choice is based on whether these one or more parameters of the problem are logically deduced from the context or explicitly expressed in speech input.

2. A method according to claim 1, additionally containing a stage, on which transform at least one task parameter according to a corresponding application programming interfaces (API) set of services the selected subset of the plurality of services.

3. A method according to claim 1 or 2, additionally containing phases in which:
collect the results of the services from the selected subset of the plurality of services; and
combine these results into a result in independent services format for the user's query.

4. A method according to claim 1 or 2, additionally containing phases in which:
evaluate the relative ability of a plurality of subsets of services many of these services to satisfy a user query, each of this set of subsets is adapted to provide functions according to the identified set of relevant PA�of amerov tasks; and
choose, on the basis of this estimation, one of the plurality of subsets of services to activate during the execution of a workflow.

5. A method according to claim 1 or 2, wherein each service is associated with a corresponding model of the characteristics of services that contains a declarative description of the relevant characteristics and properties of the service.

6. A method according to claim 5, in which selective activation of the said subset of services contains the stage at which dynamically choose from the many services a subset of a plurality of services to activate on the basis of appropriate models of characteristics of services.

7. A method according to claim 1 or 2, additionally containing phases in which:
form at least one combined outcome based on the results taken from selectively activated subset of services; and
selectively activate the second set of services for annotating, at least one merged result.

8. Electronic device for implementing intelligent automated assistant, comprising:
one or more processors;
a memory that stores instructions, wherein the instructions when performed by one or more processors, instruct the one or more processors to perform operations, comprising:
receiving a user request, and custom� request includes, at least a speech input received from the user;
processing the user request to obtain a representation of the user's intention, and this view of the intention of the user connects the user's request with a task flow implementing the requested task, wherein the task flow is adapted to activate a variety of services, each of which supports the functions according to the corresponding set of parameters of the services;
identifying, based on the submission of the intention of the user, the set of relevant task parameters for the task flow, and many of the relevant task options comprises at least one task parameter, pronounced in speech input, and at least one task parameter, logically derived from the context information associated with speech input; and
the execution flow of tasks through selective activation of a subset of a plurality of services in accordance with many of the relevant parameters of the task, wherein the execution of the task flow contains, after an unsuccessful attempt to perform a task in accordance with all of the many relevant parameters of the problem, selecting one or more parameters of the task from the set of relevant parameters of the problem to mitigate with additional try to perform the task, and the choice of osnovyvaet�Xia, were these one or more parameters of the problem are logically deduced from the context or explicitly expressed in speech input.

9. The device according to claim 8, in which the instructions, when performed by one or more processors, instruct the processors to perform any of the methods according to claims. 2-7.

10. The machine-readable medium on which is stored instructions that when performed by one or more processors, instruct the processors to perform operations:
receiving a user request, wherein the user request includes at least a speech input received from the user;
processes the user request to obtain a representation of the user's intention, and this view of the intention of the user connects the user's request with a task flow implementing the requested task, wherein the task flow is adapted to activate a variety of services, each of which supports the functions according to the corresponding set of parameters of the services;
identify, based on the submission of the intention of the user, the set of relevant task parameters for the task flow, and many of the relevant task options comprises at least one task parameter, pronounced in speech input, and at least one parameter W�cottages, logically derived from the context information associated with speech input; and
the execution flow of tasks through selective activation of a subset of a plurality of services in accordance with many of the relevant parameters of the task, wherein the execution of the task flow contains, after an unsuccessful attempt to perform a task in accordance with all of the many relevant parameters of the problem, selecting one or more parameters of the task from the set of relevant parameters of the problem to mitigate with additional try to perform the task, and this choice is based on whether these one or more parameters of the problem are logically deduced from the context or explicitly expressed in speech input.

11. Machine-readable medium according to claim 10, wherein the instructions, when performed by one or more processors, instruct the processors to perform any of the methods according to claims. 2-7.



 

Same patents:

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering. A method for electronic notary certification of text information, which includes preliminary registration of a contractor in an "electronic notary" system; when sending information by electronic mail, indicating in the "copy" field the address of the automatic "electronic notary" system; upon receiving a copy of the sent information, the automatic mail enters the copy into the personal accounts of the sender and the receiver with indication of the time of sending, the sender, the addressee of the information and all attachments comprising the information, wherein the automatic mail further notifies the addressee on the sending of information and storage of the certified copy of the sent information on the website in the personal account of the contractor.

EFFECT: confirming the sending of a document or file with indication of the date and time of sending via electronic mail.

4 cl

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to providing secure transactions in payment systems using bank cards. The technical result is the low risk of card transactions that are unauthorised by the card holder when the card is compromised. In the method of conducting authorised card transactions, the card holder changes said card parameters in advance to permitting parameters by sending an authenticated request to a processing centre of the issuer, thereby permitting one or more authorised transactions; an authorised transaction can be approved by the processing centre of the issuer only if card parameters permitting said transaction were established by an authentication request by the card holder.

EFFECT: present solution minimises the risk of transactions that are unauthorised by a card holder in any environment of using cards and is distinguished by the capability for use in existing systems for controlling authorised transactions.

3 cl, 5 dwg

FIELD: physics, computation hardware.

SUBSTANCE: invention relates to creation and operation of data base of vacancies and manpower resources. Search system for every selected data element defines its rating and processes data of data base and includes structuring in relational view of data on structure, terms and interrelations as to official characteristics with allowance for taxonomy and ontology of subject area as well as compilation of the rating of indices making the matrix and their ranging. Clients-physical persons define the priorities of criteria for fulfilled job and selection of a company. Besides, they enter the obligatory magnitude of criteria and varying ranged indices for selection of vacancies to be automatically processed by the system in scoring mode. For this, these are ganged after determination of their mean arithmetic value. Client-physical person defines the obligatory values of criteria and varying ranged indices for personnel selection. The latter are processed by the system in scoring mode. For this all candidates selected in compliance with used complex of search criterion after definition of their mean arithmetic value get the rating in the list of resumes of selected candidates for client-physical person the make the decision on.

EFFECT: accelerated search at ranging of data on personnel or vacancies.

8 cl, 1 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to the field of optimisation of selection and delivery of products in a trading network. The system for the network optimisation implemented by the computer comprises: the first processor for data processing for data receiving and storage in data memory; the second processor for data processing for obtaining at least one input data for conditional dynamic options in a network; the third processor for data processing for obtaining at least one input data set for the specified network to define the specified chosen products with use of at least one optimised filters; the fourth processor for data processing for delivery of at least one of the named products at least to one of the named organisations participating in a trade network and the fifth processor for data processing for data recording relating to the named delivered products, in the data memory.

EFFECT: optimisation of selection of a product or service with reference to requirements of clients and ensuring dynamic integration of these requirements.

16 cl, 5 dwg

FIELD: physics, control.

SUBSTANCE: invention relates to computer engineering. A climate control device, having a network interface configured to receive a request to use the climate control device; transmit an authorisation request to a payment system in response to receiving said request; receive an activation message containing information associated with the payment system in response to the authorisation request; transmit a payment message to the payment system based on a plurality of usage parameters, wherein the payment message leads to the deduction of funds from the account of the user; a user interface configured to receive user input which identifies a climate control programme containing a temperature parameter, a humidity parameter, an ionisation parameter, a dryer parameter, an air purification parameter, a sound parameter and a fragrance parameter, wherein each of said parameters is associated with a corresponding one of a plurality of times, a plurality of durations and a plurality of set values; and a climate control subsystem configured to change the environment based on the climate control programme.

EFFECT: automated climate control on a confined territory.

13 cl, 1 tbl, 11 dwg

FIELD: medicine.

SUBSTANCE: invention relates to transfusiology and is intended for application in medical institutions in planning and transfusion to patient of preserved donated blood components. Method is realised with application of bar-coding technology, with formation of unified information and technological system, where databank of patients who have transfusions of donated blood components in anamnesis is maintained; transfusion plan, with respect to which available components of donated blood are estimated, is composed in accordance with operation plan, and request is sent to blood centre for missing volume of donated blood components, data about obtained blood components are automatically included into medical information system.

EFFECT: ensuring safety of transfusion of donated blood components.

7 cl, 18 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to activation of services using algorithmically configured keys. The method of user subscription to the service comprises: identification in the computer of issuer of the user who is authorized for subscription to a service on the basis of a criteria determined by an issuer; extraction by a computer of the issuer of data associated with the user and shared data element which is shared by the issuer computer and the service provider computer; formation of the first activation code by the issuer computer and sending the first activation code to the user; and the user sends the first activation code and data associated with the user to the service provider computer; and the service provider computer forms the second activation code and authorizes the user for subscription to service, if the first and the second activation codes are identical.

EFFECT: prevention of violation of security of data processing system.

20 cl, 9 dwg

FIELD: medicine.

SUBSTANCE: invention relates to medical working process of visualisation. System contains: control unit (114), containing processor (116); and multitude of processing executives (102), connected with it, with processor (116) realising control of realisation with feedback of plane of medical visualisation working process by means of multitude of processing executives (102), and processor (116) is made with possibility of inquiring electronic medical information, dealing with and/or not dealing with visualisation from one or more processing executives; formation of plan of visualisation procedure working process by means of visualisation procedure, based on information inquired; reception of signal, indicating due date of planned visualisation procedure; downloading visualisation protocol, corresponding to visualisation procedure, into visualisation system, used for visualisation procedure; realisation of scanning by means of visualisation system; and updating plan of visualisation procedure working process.

EFFECT: increased reliability of patient diagnostics.

15 cl, 9 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to systems and methods of conducting a transaction to acquire available goods or services using stored means associated with designation. The methods include: receiving, at a point of sale, identifiers of goods or services selected for acquisition, and designation; determining if the goods or services comprise available goods or services; calculating the sum required to acquire the available goods or services; determining if the stored means are enough to acquire the available goods or services; using the stored means to acquire the available goods or services; and using an additional funding source provided by the customer to acquire any goods or services which are not available goods or services. The system implements said methods.

EFFECT: high accuracy and reliability of a transaction, which includes an additional alternative source.

18 cl, 9 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to a computer-readable data medium of a web configuration system, a method of customising a welding system and a web configuration system for customising a welding system. The web configuration system includes at least one processor and one or more tangible, computer-readable media at least collectively including or storing instructions executable by the at least one processor. The instructions include instructions to limit configuration options for a welding system based on user selection of a power source for the welding system, instructions to calculate a weld cable size for the welding system based on the user selection of the power source and user selection of input voltage, instructions to assign weld cable characteristics based on the weld cable size and a user selected cable length, and instructions to assemble a welding system order based at least on the user selection of the power source and the weld cable characteristics.

EFFECT: obtaining a computer-readable data medium of a web configuration system.

20 cl, 10 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to providing a user terminal with an application which is customised with respect to user-specific parameters and terminal-specific properties. An application server stores a template for the customised application of an identification document and a feature set for each of several terminal types; receives a request to generate a customised application of the identification document for a user and determines individual user parameters based on the request; transmits, to the user terminal, a message with data which initiates the request from the user terminal; determines the type of the user terminal based on the request; determines individual properties of the terminal based on the type of the user terminal; generates a customised application of the identification document based on the template, individual user parameters and individual terminal parameters; inserts the customised application of the identification document into the delivery file set of a data transmission system and transmits the file set for delivery to the user terminal.

EFFECT: providing a customised application for downloading in a user terminal.

8 cl, 5 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer networks, namely to client-server computer networks. The technical result is achieved due to the use of pre-processing on the server for expansion of static views of electronic documents in a computer network. The electronic document can be published from the client computer on the first server computer in a computer network. During the publication the first server computer can create static view of the electronic document and the application for viewing of static view of the electronic document on the client computer. The first server computer can keep static view of the electronic document and the viewing appendix on the second server computer for delivery to the client computer.

EFFECT: increase in network productivity and minimising of delay of delivery of electronic documents requested by users.

7 cl, 5 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to means of executing services on a server and a client of a client-server architecture. When making a user decision at a predefined point during the process of executing services, execution of services is suspended at the server and a user decision request is sent to the client. The user decision request includes information which requests the user to make a decision with respect to executing services after the user decision making point. After receiving information on the user decision generated by the client in response to the user decision request, the method includes determining action on executing services, which corresponds to the received information on the user decision, based on the corresponding link between information on the user decision and an instruction to execute services. Services are executed in accordance with the determined action on executing services.

EFFECT: enabling change in the sequence of actions on executing services on a server.

11 cl, 6 dwg

FIELD: physics, computation hardware.

SUBSTANCE: invention relates to antivirus software and hardware, particularly to emulator state retention and its further recovery. File is received for emulation. Check for first-time emulation is executed. Emulator state image is defined including at least the image of emulating system to be loaded in emulator for further file emulation. File is emulated. Emulator state images are created. Note here that every said image comprises at least the image of emulating system. Incorrect file emulation termination is tested. Required emulator state image is selected for continuation of emulation in the case of incorrect file emulation termination. Selected emulator state image is loaded for file emulation continuation.

EFFECT: accelerated emulation.

14 cl, 6 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to the method and system for distribution of changes in the technical equipment configuration. The method for covering by changes of the technical equipment configuration by means of transfer of the modified configuration of the original in which all or some changes which have been made in the original, are spread from the original to the named, at least, one copy by semi-automatic or completely automatically method, and possible conflicts are indicated automatically by means of graphic or text display, and covering by changes from the original comprises three following stages: a) agreeing at which the original and the copy or copies are analysed to identify, which objects in the copy have been copied from which object from the original and, therefore, correspond to each other, b) comparison at which various types of changes are considered with checking for the possible conflicts, and c) synchronisation at which the user makes the decision, which changes are applicable to the copy for synchronisation of the copy with the original.

EFFECT: increase of level of automation of system of coverage by changes at simultaneous minimising of conflicts.

19 cl, 6 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to intelligent automated assistant systems. Disclosed is method of operating an intelligent automated assistant. The method is carried out in an electronic device having a processor and memory which stores instructions for execution by the processor. The processor executes instructions on which a user request is received, wherein the user request includes a speech input received from the user. A prompt is provided to the user, the prompt presenting two or more properties relevant to items of an object selection domain. The user is requested to specify relative importance between the two or more properties.

EFFECT: high accuracy of providing a user with relevant information owing to consideration of relative importance between properties which correspond to items of an object domain.

12 cl, 50 dwg, 5 tbl

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to intelligent automated assistant systems. Disclosed is method of operating an intelligent automated assistant. The method is carried out in an electronic device having a processor and memory which stores instructions for execution by the processor. The processor executes instructions on which a user request is received, wherein the user request includes a speech input received from the user. Two or more alternative interpretations of user intent are obtained based on the received user request and one or more similarities and one or more differences between said alternatives are identified. Further, the user is presented with a response, said response being at least one of the identified differences.

EFFECT: high accuracy of presenting relevant interpretations of user intent in the correct context.

13 cl, 50 dwg, 5 tbl

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to electronic notebook systems. Disclosed is a computer- implemented job scheduling system. The system includes at least one computer which executes a job tracking component associated with a shared electronic notebook for creating and maintaining tracking information of jobs processed with respect to shared notebook information. The shared electronic notebook has a root directory containing multiple files and directories, wherein sections of the notebook are associated with files and folders of the notebook are associated with directories. Tracking information is stored as a job file in the root directory of the shared electronic notebook and has properties relating to the user-selected change with respect to at least part of the file in the shared electronic notebook.

EFFECT: automating scheduling for synchronisation of a new job through a client instance depending on information of the shared notebook, based on tracking information.

14 cl, 11 dwg

FIELD: physics.

SUBSTANCE: method of functioning of a smart automated assistant is offered. The method is performed in the electronic device containing the processor and memory where the instructions executed by the processor are saved. The processor executes instructions by which the user request is accepted, which includes the speech input accepted from the user. The information on the sender name is taken from the transfer accepted in the electronic device before reception of speech input. Meanwhile this transfer is accepted from the sender that is isolated from the mentioned user. The intention of the user is revealed on the basis of the mentioned text line and a sender name.

EFFECT: improvement of accuracy of representation of relevant information to the user due to identification of intention of the user on the basis of the text line and a sender isolated from a user.

15 cl, 50 dwg, 5 tbl

FIELD: physics, computer engineering.

SUBSTANCE: group of inventions relates to computer engineering and can be used for setting storage keys used in a computing environment. Disclosed is a means of setting a key without switching to passive state, which enables to manipulate storage keys without switching to passive state operations of other processors of a multiprocessor system. Using said means, a storage key available for a plurality of processors of a multiprocessor system is updated without switching to passive state operations of the plurality of processors. Since the storage key is updated without switching to passive state the rest of the operations, the processor can recognise the storage key as having one value at the beginning of the operation and a second value at the end of the operation. A mechanism is provided, which enables to continue an operation to avoid a fatal condition.

EFFECT: high efficiency.

8 cl, 22 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to intelligent automated assistant systems. Disclosed is method of operating an intelligent automated assistant. The method is carried out in an electronic device having a processor and memory which stores instructions for execution by the processor. The processor executes instructions on which a user request is received, wherein the user request includes a speech input received from the user. A prompt is provided to the user, the prompt presenting two or more properties relevant to items of an object selection domain. The user is requested to specify relative importance between the two or more properties.

EFFECT: high accuracy of providing a user with relevant information owing to consideration of relative importance between properties which correspond to items of an object domain.

12 cl, 50 dwg, 5 tbl

Up!