Image processing device and method

FIELD: physics, photography.

SUBSTANCE: invention relates to an image processing device and method, which can improve encoding efficiency, thereby preventing increase in load. The technical result is achieved due to that a selection scheme 71 from a prediction scheme 64 by filtering selects a motion compensation image for generating a prediction image at a high-resolution extension level from key frames at a low-resolution base level. The filter scheme 72 of the prediction scheme 64 by filtering performs filtration, which includes high-frequency conversion and which uses analysis in the time direction of a plurality of motion compensation images at the base level, selected by the selection scheme 71, in order to generate a prediction image at the extension level.

EFFECT: reducing load in terms of the amount of processing owing to spatial increase in sampling frequency at the base level for encoding the current frame.

19 cl, 26 dwg

 

The technical field to which the invention relates.

The invention relates to a device and method of image processing, and, more particularly, to a device and method of image processing, which can improve the coding efficiency by preventing the increase of the load.

The level of technology

When processing moving images are commonly used coding schemes that employ motion compensation and orthogonal transformation such as discrete cosine transformation, the transformation of karunen-Loev or the wavelet transform, which includes MPEG (Expert group on the moving image), N. H, etc., In such schemes, coding of moving images, reducing the amount of code is achieved by using the correlation in the spatial direction and the time direction among the characteristics of the input image signal intended for encoding.

For example, in N. 264, unidirectional prediction or bidirectional prediction is used when the intermediate frame, the frame is subjected to inter-frame prediction (inter prediction), generated using the correlation in the time direction. Interframe prediction develop for generating image prediction-based frames in R the value of time.

In addition, SVC (scalable video encoding), which is an extension of the standard N. 264, was established coding scheme, which takes into account the spatial scalability. SVC (H. 264/AVC Annex G) is a modern standard of coding, which was standardized in November 2007, ITU-T (international telecommunication Union - telecommunication Sector) and ISO/IEC (international organization for standardization/international electrotechnical Commission).

In Fig.1 illustrates the relationship of links to image-forming prediction for compression, which take into account the spatial scalability in SVC. In SVC encoding is performed to set the resolution, for example, at the Foundation level and at the level of improvement shown in Fig.1. In the case of the example according to Fig.1, as a base, encode an image having a resolution of n×m [pixels (pix)] (n and m are whole numbers) using spatial scalability. Along with this, the image having a resolution of N×M pixels (pix)] (N and M are integers, where N>n, and M>m), as the extension level, encode, using spatial scalability.

In the case of Foundation level encode the current frame using intra-frame prediction or inter-frame prediction is nelogicno the occasion of encoding based on the standard N. 264. In the case of the example shown in Fig.1, when performing encoding on the Foundation layer, use two reference planes (Ref0, Ref1). From a separate reference planes emit image (MS, MS) motion compensation, and performing interframe prediction.

Also, in the case of the extension level, as in the case of the Foundation level, the current frame may be encoded using intra-frame prediction or inter-frame prediction.

In the case of the intraframe prediction prediction performed using spatial correlation on the extension level of the current frame. Intra-frame prediction is effective for the moving image to be encoded, when the correlation in the time direction is low, for example, when a subject moves a short distance. However, in the conventional moving images, in many cases, the correlation in the time direction is higher than the prediction in the spatial direction, and intra-frame prediction is not optimal from the point of view of coding efficiency.

In the case of the interframe prediction decoded image on the extension level temporarily preceding or following frames are used as reference planes. When interframe prediction using the correlation in n the Board time and thus, make feasible a high coding efficiency. However, it is required that you had to pre-decode the image frame with high resolution on the extension level, which are used as a reference plane. In addition, you must also save the high resolution image in a storage device for use as reference images. In addition, you need to read high-resolution images with a large amount of data from the storage device. In accordance with this interframe prediction can be called a scheme, which imposes a large burden from the point of view of the amount of processing and cost of implementation.

In this regard, in the case of the extension level, in addition to the above two schemes, you can use the method of forecasting based on spatial increase in the frequency of sampling (conversion) frequency of Foundation level (below called a prediction conversion with increasing frequency) to encode the current frame.

The image on the Foundation level is a version of the low resolution image at the level of expansion, and, therefore, it can be considered as comprising a signal corresponding components of the low frequency image on ur the outside of the extension. That is, the image on the degree of expansion can be obtained by adding the high-frequency components to the image at the Foundation level. Forecasting with a transform with increasing frequency is a way to perform prediction using the correlation between levels, and is a method of forecasting that are useful for improving coding efficiency, in particular, in the case where intra - or inter-frame prediction is not applied. In addition, this method of forecasting decodes the image on the extension level for the current frame by decoding images simultaneously on the Foundation level, and, therefore, it is possible to tell, this prediction scheme is great (which puts a small load) from the point of view of the amount of processing.

At the same time, the processes for increasing the resolution include the technology to perform the motion compensation and filtering FIR pixel values, to transform the correlation in the time direction in the spatial resolution for use (see, for example, NPL 1).

In the method described in NPL 1, the correlation in the time direction is used to process aimed at increasing the resolution of the input image sequence. In particular, expect information about the different the minute forecast/compensated by the movement of the image between the current image and the previous image and serves, as feedback, the target current image to restore the high-frequency component included in the input image.

References

Non-patent literature

NPL 1: "Improving Resolution by Image Registration", MICHAL IRANI AND SHMUEL PELEG. Department of Computer Science, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel Communicated by Rama Chellapa, Received June 16, 1989; accepted May 25, 1990.

The invention

Technical task

However, the prediction conversion with increasing frequency image with the lower resolution results in the image having a small number of high-frequency components generated due to the influence of the linear interpolation filter. When the prediction conversion with increasing frequency, therefore, can only be obtained picture prediction with a small amount of high-frequency components. That is, when the prediction conversion with increasing frequency is not to say that the transmitted information of a pixel on the Foundation level sufficiently used to perform prediction. On the extension level, therefore, a large amount of code may be required for encoding the residual signal.

As noted above, in conventional methods of encoding and decoding was difficult to achieve at the same time and improve coding efficiency, and prevent the led is placed load.

Thus, can be considered the manner in which the improvement of the encoding efficiency is realized by applying the image processing techniques, as described in NPL 1, to transform the temporal correlation of the moving image in spatial resolution. However, the method described in NPL 1, you can't just apply for SVC.

For example, when interframe prediction, the resolution of the image motion compensation received from the reference plane is the same as the image prediction, which generate, and the method described in NPL 1, cannot be used to convert the prediction with increasing frequency. In addition, when the prediction conversion with increasing frequency, since the image prediction generated only from the image of the current frame on the Foundation level, the method described in NPL 1, where resolution increases, using three images, cannot be used to predict conversion with increasing frequency.

The present invention has been proposed considering the above situation, and it is intended to do the encoding, which takes into account spatial scalability through more effective use of temporal correlation is included in the sequence of signals in a moving image, ensuring that the same is by way the possibility of improving coding efficiency by preventing the increase of the load on processes such as encoding and decoding.

The solution of the problem

In one aspect the present invention provides an imaging device, comprising: a decoder designed to decode the encoded image; generating tool that is intended to be a summation of the image decoded by the decoding means, and the image prediction, and to generate a decoded image; an allocator that is intended to perform motion compensation using, as reference frames, frames formed of the decoded images generated by the generating tool, and using the motion vectors in the images that have been encoded, and for selecting image motion compensation having a lower resolution than the image prediction, from the reference frames corresponding to the image prediction; and a means of generating image prediction, designed to perform a filtering process for compensating for image motion compensation allocated by the allocator in relation to the high-frequency components, using the correlation in the time direction, which VK is uchena in image motion compensation, generating, thus, the image prediction with a higher resolution than the image motion compensation.

The encoded image was hierarchically decomposed into image in multiple levels, with different resolutions, and these images were encoded; decoding means may decode the encoded image at each level; generating tool can generate the decoded image at each level; when the decoding level with high resolution, the allocator can be used as reference frames, frames at a level having a lower resolution than on the level, and highlight image motion compensation from a reference frame at a level having a lower resolution; and a means of generating image prediction can generate the image prediction at the level of the high resolution by performing a filtering process of the image motion compensation allocated from the keyframes on the level with a lower resolution.

A means of generating image prediction may include: a means of converting the resolution, designed for converting the resolution difference image between multiple image motion compensation, the dedicated tool selection and resolution enhancement; the means of the first filter designed to filter low frequency to a Delta image, whose resolution has been increased by means of resolution conversion; a second filter means designed to apply high-pass filter to an image obtained by applying the filter of low frequencies in the medium of the first filter; and means summation intended for summation image obtained by applying the filter of low frequencies by means of the first filter, and the image obtained by applying a high-pass filter in the second filter means to one of the numerous image motion compensation allocated by the allocator, and to generate image prediction.

Means summation can summarize the image obtained by applying the filter of low frequencies by means of the first filter, and the image obtained by applying a high-pass filter by means of the second filter with image motion compensation, selected from the previous frame relative to the time-image prediction.

The imaging device may further include a means unidirectional prediction, designed to perform unidirectional predicted the financing, using a variety of image motion compensation, and to generate image prediction; means bidirectional prediction, designed to perform bi-directional prediction using a variety of image motion compensation, and to generate image prediction; and means for determining intended to determine whether to generate an image prediction through unidirectional prediction means unidirectional prediction, generated through bidirectional prediction means bidirectional prediction, or generate through a process of filtering means for generating image prediction by using the flag identification included in the header of the encoded image.

In one aspect the present invention also provides a method for image processing, in which decodes the encoded image;

summarize the decoded image and an image prediction and generate a decoded image; performing motion compensation using, as reference frames, frames formed of the generated decoded image using the motion vectors in the images that have been encoded, and produce from the said motion compensation, having a lower resolution than the image prediction from the reference frames corresponding to the image prediction; and performing the filtering processing to compensate for the selected image motion compensation in respect of the high-frequency components, using a correlation in a time direction that is included in image motion compensation, generating, thus, the image prediction, which has a higher resolution than the image motion compensation.

In another aspect of the present izobreteniya provided by the imaging device, comprising: a means of encoding used to encode the original image that is an image used to encode and to generate an encoded image; detection means designed to detect the motion vectors on the basis of images and the original image, each image obtained by performing local decoding on the basis of a residual signal indicating a difference between the original image and the image prediction; allocator that is intended to perform motion compensation using, as reference frames, frames formed of images, obtained done the research Institute for local decoding and using the motion vectors, detected by the detection means, and for selecting image motion compensation having a lower resolution than the image prediction from the reference frames corresponding to the image prediction; and generating tool that is designed to perform the filtering process to compensate for image motion compensation allocated by the allocator in relation to the high-frequency components, using a correlation in a time direction that is included in image motion compensation, thereby generating the image prediction with a higher resolution than the image motion compensation.

The encoding means may generate the encoded images on multiple levels, with different resolutions; when the decoding level with high resolution, the allocator can be used as reference frames, frames at a level having a lower resolution than on the level, and highlight image motion compensation from a reference frame at the level of having a low resolution, using the motion vectors detected by the detection means at a level having a lower resolution; and generating tool can generate the image prediction at the level of the high resolution, polnaya the filtering process for the image motion compensation, extracted from keyframes, on the level with low resolution.

A means of generating may include a means of converting the resolution, designed for converting the resolution difference image between multiple image motion compensation allocated by the allocator, and increasing the resolution; the means of the first filter designed to filter low frequency to a Delta image, whose resolution has been increased by means of resolution conversion; a second filter means designed to apply high-pass filter to an image obtained by applying the filter of low frequencies means of the first filter; and means summation intended for summation image obtained by applying the filter of low frequencies by means of the first filter, and images obtained by applying a high-pass filter by means of the second filter, for one of the many image motion compensation allocated by the allocator, and generating image prediction.

Means summation can summarize the image obtained by applying the filter of low frequencies by means of the first filter, and the image obtained by applying a high-pass filter means Vtorov the filter with image motion compensation, selected from the previous frame relative to the time-image prediction.

The encoding means can ensure the header flag identification that identifies whether the image prediction that will be added to the image decoded by the decoding device, to be generated through unidirectional prediction, generated through bidirectional prediction, or generated through the filtering process.

In another aspect of the present invention also provides a method for image processing, which encode the original image that is an image used to encode and generate the encoded image; detects motion vectors based on the images and the original image, and each of the images get by performing local decoding on the basis of a residual signal indicating a difference between the original image and the image prediction; performing motion compensation using, as reference frames, frames formed of images obtained by performing local decoding and using the detected motion vectors, as well as highlighting the image motion compensation having a lower resolution than the image the response prediction from the reference frame, relevant images forecasting; and performing the filtering processing to compensate for the selected image motion compensation in respect of the high-frequency components, using a correlation in a time direction that is included in image motion compensation, thereby generating the image prediction with a higher resolution than the image motion compensation.

In one aspect of the present invention decodes the encoded image; a decoded image and an image prediction summarize and generate a decoded image; performing motion compensation using, as reference frames, frames formed of the generated decoded image, and using the motion vectors in the images that have been encoded, and produce image motion compensation having a lower resolution than the image prediction from the reference frames corresponding to the image prediction; and perform processing filter designed to compensate for the selected image motion compensation in respect of the high-frequency components, using the correlation in the time direction, which is included in image motion compensation, generating, thus, the image prediction, that is has a higher resolution, than the image motion compensation.

In another aspect of the present invention encode the original image that is an image used to encode and generate the encoded image; the motion vectors detected based on the images and the original image of each image obtained by performing local decoding on the basis of a residual signal indicating a difference between the original image and the image prediction; perform motion compensation using, as reference frames, frames formed of images obtained by performing local decoding and using the detected motion vector, and image motion compensation having a lower resolution than the image prediction, separated from the reference frames corresponding to the image prediction; and perform processing filter designed to compensate for the selected image motion compensation in respect of the high-frequency components, using a correlation in a time direction that is included in image motion compensation, thereby generating the image prediction with a higher resolution than the image motion compensation.

Preferred effects invented the I

Information can be processed in accordance with the present invention. In particular, the image prediction with high accuracy can be generated, and the coding efficiency can be improved without increasing the load more than necessary.

Brief description of drawings

In Fig.1 shows a diagram illustrating the decoding of the coding scheme, which takes into account the usual spatial scalability.

In Fig.2 shows a diagram representing an overview of the generation image prediction, in which the present invention is applied.

In Fig.3 shows a block diagram illustrating an example configuration of the main part of the decoding device, in which is applied the present invention.

In Fig.4 shows a block diagram illustrating an example configuration of the main part of the decoding scheme, without loss Fig.3.

In Fig.5 shows a block diagram illustrating an example configuration of a main part schematic prediction/motion compensation according to Fig.3.

In Fig.6 shows a block diagram illustrating an example configuration of the main part of the scheme of the prediction filter by Fig.5.

In Fig.7 shows a block diagram of a sequence of operations illustrating an example flow of the decoding process.

In Fig.8 shows a block diagram of a sequence of operations, p is Yasnaya example of the flow of the decoding process without losses.

In Fig.9 shows a block diagram of a sequence of operations illustrating an example flow forecasting process by filtering when performing the decoding.

In Fig.10 shows a block diagram illustrating an example configuration of the main part of the encoder, in which is applied the present invention.

In Fig.11 shows a block diagram illustrating an example configuration of the main part of the schema definition of the mode in Fig.10.

In Fig.12 shows a block diagram illustrating an example configuration of a main part schematic prediction/motion compensation.

In Fig.13 shows a block diagram of a sequence of operations illustrating an example flow of the encoding process.

In Fig.14 shows a block diagram of a sequence of operations illustrating an example of a process flow definition mode.

In Fig.15 shows a block diagram of a sequence of operations illustrating an example flow forecasting process by filtering when performing the encoding.

In Fig.16 shows a diagram illustrating another example of a brief overview of the decoding process applied to the present invention.

In Fig.17 is a block diagram illustrating another example configuration of the filter of Fig.6.

In Fig.18 is a diagram explaining still another example of the overview of the decoding process, which is applied the present from Britanie.

In Fig.19 shows a flowchart of the sequence of operations illustrating another example of process flow forecasting by filtering when performing the decoding.

In Fig.20 shows the block diagram of the sequence of operations illustrating another example of the forecasting process by filtering when performing the encoding.

In Fig.21 shows a block diagram illustrating an example configuration of a main part of a personal computer that utilizes the present invention.

In Fig.22 shows a block diagram illustrating an example configuration of a main part of a television receiver to which is applied the present invention.

In Fig.23 shows a block diagram illustrating an example configuration of a main part of a mobile phone that utilizes the present invention.

In Fig.24 shows a block diagram illustrating an example configuration of a main part of a recording device, a hard disk, in which is applied the present invention.

In Fig.25 shows a block diagram illustrating an example configuration of the main part of the chamber in which is applied the present invention.

In Fig.26 shows a diagram illustrating examples of the size of the macroblock.

Detailed description of the invention

Further explains the modes of carrying out the invention (below referred to as variants of implementation). Should be about the mark, that explanation will be presented in the following order:

1. The first version of the implementation (decoding processing)

2. The second version of the implementation (coding)

3. A third option implementation (processing decoding with three or more image motion compensation)

4. The fourth option implementation (processing of decoding and processing coding using image motion compensation at the same level),

1. The first option exercise

Overview of forecasting

In Fig.2 shows a diagram illustrating the overview of a method of generating image prediction, which uses the present invention. As is shown in Fig.2, in this case, filtering is performed to image multiple reference planes at the Foundation level, to generate image prediction of the current block (processing target block in the current time) on the extension level.

Thus, using the analysis in the time direction, it becomes possible to more effectively use the components of the signal in the sequence of images than using a spatial filter with the increase of the sampling rate. As a result, the image prediction generated using technology in accordance with the present invention (below called the prognosis is m filter), can reduce the residual elements of forecasting with spatial high-frequency components than the image prediction generated using conventional forecasting with the conversion of the sampling rate, which uses the image of the current frame processing target frame at the current time) in the base layer. That is, the amount of code for the image intended for coding at the level of expansion can be reduced, and thus it is possible to improve coding efficiency.

In addition, this prediction filtering does not apply to the decoded images at the level of expansion in different time frames. Thus, the amount of processing required for coding, the capacity for temporary storage, the amount of information read from the storage device, etc. can be reduced, and the costs of the embodiment can be reduced. In addition, you can also reduce energy consumption.

Device configuration decode

In Fig.3 shows a block diagram illustrating an exemplary configuration of the decoding device 1 according to the embodiment of the present invention.

Information about the image encoded by the encoding device, described below, which indicate to the decoding device 1 via a cable, network or removable media. Examples of information of the compressed image includes information of the image, encoded in accordance with the standard H. 264/SVC.

In SVC compressed image information consists of many levels of resolution. The level with the lowest resolution is a Foundation level, and the levels with higher resolution than the Foundation level, represent the levels of expansion. It should be noted that the number of levels is arbitrary; however, in the following description it is assumed that information of the compressed image consists of two levels. Thus, the compressed image information supplied to the decoding device 1, has the Foundation level and one level of expansion.

The compressed image information of each frame is sequentially fed into the decoding device 1. In each frame sequentially enter the bit streams corresponding levels from low resolution to the side with high resolution. That is, the bit stream at the Foundation level previously served in the decoding device 1.

The bit stream on the Foundation level decode as in the case of information of the compressed image based on the standard H. 264/AVC, and its explanation is not presented here. After the bit stream at the Foundation level is decoded, the bit stream is and the level of expansion is served in the decoding device 1. Basically, the processing of the bit stream on the extension level is illustrated below.

In the buffer 11 save consistently maintain the flow of bits as input information of the compressed image. The information contained in the buffer 11 save, read through circuit 12 decodes losslessly in units of images of a particular dimension, such as macroblocks forming the frame accordingly. In the standard N. 264 it is also possible to perform processing instead of units of macroblocks of 16×16 pixels, in units of blocks into which a macroblock is additionally divided, such as blocks of 8×8 pixels or 4×4 pixels.

Scheme 12 decode lossless performs decoding processing corresponding to the encoding scheme, such as the processing of decoding variable length or the processing of an arithmetic decoding of the image read from the buffer 11 save. Scheme 12 decode lossless outputs the quantized conversion coefficient, which is obtained by executing the decoding process, in scheme 13 dekvantovanie.

In addition, the circuit 12 decodes lossless identifies the method of forecasting based on the flag identification included in the image header, intended for decoding. In case, when determining that the image is intended for decoding, p is ecstasy an image with the intra-frame coding, scheme 12 decode lossless displays information mode intraframe prediction stored in the image header, the circuit 22 intra-frame prediction. Information mode intraframe prediction includes information related to the intra-frame prediction such as block size, used as a unit during processing.

In case, when determining that the image is intended for decoding, this is the information interframe coding scheme 12 decode lossless outputs the motion vector and flag identification contained in the image header, the circuit 21 prediction/motion compensation. The prediction mode, in which the image prediction should be generated using interframe prediction, identify, using the flag identification. Flags identify set in units of, for example, macroblock or frame.

In addition to the mode unidirectional prediction mode of bidirectional prediction and the prediction mode conversion with increasing frequency, prepared the prediction modes include a prediction mode by filtering to generate image prediction by performing the filtering depicts the th compensated with movement, selected from a set of reference frames located in one or both of the temporary directories on the Foundation level.

Below is a prediction mode in which the pixel value in one image motion compensation among image motion compensation, selected from a set of reference frames located in one direction, is set as the pixel value in the image prediction, simply called unidirectional prediction. In addition, the prediction mode, in which the average value of pixel values in the image motion compensation, individually selected from the set of reference frames located in both directions, set as the pixel value in the image prediction, simply called bidirectional prediction. In addition, the prediction mode, in which the image motion compensation allocated from the current frame on the Foundation level, transform with increasing frequency to determine pixel values in the image prediction mode is simply forecasting with a transform with increasing frequency.

The fourth prediction mode, as shown in Fig.2, in which the pixel value in the image prediction is determined by the filtering process, which includes prognosisof is compared with the conversion frequency for each image motion compensation, selected from a set of reference frames located in one or both directions of the Foundation level, called prediction by filtering.

Scheme 13 dekvantovanie performs dekvantovanie quantized transform coefficients supplied from the circuit 12 decodes lossless, using a scheme corresponding to the quantization scheme used on the side of coding. Scheme 13 dekvantovanie outputs the conversion coefficient obtained by performing dekvantovanie in scheme 14 inverse orthogonal transformation.

Scheme 14 the inverse orthogonal transform performs, for example, the inverse orthogonal transform of the fourth order with respect to the conversion factor supplied from the circuit 13 dekvantovanie using the schema corresponding to the schema of an orthogonal transformation, on the side of coding, such as discrete cosine transformation or conversion of karunen-Loev, and displays the resulting image in scheme 15 adder.

Scheme 15 adder combines the decoded image filed from the circuit 14 inverse orthogonal transform, and the image prediction transferred from the circuit 21 prediction/motion compensation, or from the circuit 22 intraframe prediction through the switch 23, and outputs the composite the images into the filter 16 of the address block.

The filter 16 address blocks removes noise blocks included in the image transmitted from the circuit 15 of the adder, and outputs an image from which you have removed the noise block. Image output from the filter 16 address blocks, served in the buffer 17 changes the layout in the storage device 19 of the frame.

The buffer 17 changes the layout temporarily stores the image transferred from the filter 16 address blocks. The buffer 17 changes the layout generates individual frame of the image, for example, each macroblock, which retains and modifies the layout of the generated frames in a specific order, such as the display order, before displaying them in scheme 18 D/A (digital to analog) Converter.

Scheme 18 D/A Converter performs D/A conversion of each of the frames supplied from the buffer 17 changes the layout, and displays these signals in the form of frames out.

The storage device 19 of the frame temporarily stores the image transferred from the filter 16 address blocks. The information stored in the storage device 19 of the frame, serves in scheme 21 prediction/motion compensation or circuit 22 intraframe prediction through the switch 20. It should be noted that the storage device 19 of the frame also contains an image at the Foundation level, which has been decoded before decoder what level of expansion, and the saved image is used to decode the extension level, as described below.

The switch 20 is connected to the output A1 in the case when the image prediction should be generated using interframe prediction, and is connected to the output b1, in the case when the image prediction should be generated through intra-frame prediction. By switching the switch 20 is controlled, for example, using the circuit 31 of the control.

Scheme 21 prediction/motion compensation determines the prediction mode in accordance with the flag of identification supplied from the circuit 12 decodes lossless, and selects frames intended for use as reference frames, among the decoded frames are stored in a storage device 19 of the frame in accordance with the mode of prediction. Scheme 21 prediction/motion compensation determines the macroblocks corresponding to the target image prediction, among macroblocks forming the keyframes on the basis of the motion vectors supplied from the circuit 12 decodes lossless, and determines the macroblock, as the image motion compensation. Scheme 21 prediction/motion compensation determines the value of a pixel in the image prediction of pixel values in the images compensatively, in accordance with the prediction mode, and displays the image prediction pixel value which has been defined in the schema adder 15 through the switch 23.

Scheme 22 intraframe prediction performs intra-frame prediction in accordance with the information of the intra-frame mode prediction transmitted from the circuit 12 decodes lossless, and generates the image prediction. Scheme 22 intraframe prediction outputs the generated image prediction scheme adder 15 through the switch 23.

The switch 23 connected to the output A2 in the case when the image prediction was generated using the schema 21 prediction/motion compensation, and combined with the output of b2 in the case when the image prediction was generated by the circuit 22 intra-frame prediction. Switching switch 23 also controls, for example, the circuit 31 of the control.

Circuit 31 controls the overall operation of the decoding device 1, for example, by switching the connection of the switches 20 and 23. Identification method of prediction for image processing purposes can be performed by a circuit 31 of the control.

In Fig.4 shows a block diagram illustrating an example configuration of the main part of the circuit 12 decodes losslessly in Fig.3

As shown in Fig.4, the circuit 12 decodes lossless includes circuit 41 definition of forecasting and circuit 42 decode processing. Scheme 41 definitions forecasting determines how the prediction image supplied from the buffer 11 save. Scheme 41 determine the prediction identifies a method for predicting, based on, for example, flag identification included in the image header, intended for decoding. It should be noted that the circuit 41 definition of forecasting, of course, can identify the method of forecasting by analyzing the bit stream. In this case, the flag identification can be eliminated, and the amount of information to information of the compressed image can be reduced.

In case, when determining that the image is intended for decoding, is an image with intraframe coding scheme 41 determine the forecasting outputs information of the intra-frame mode prediction stored in the image header, the circuit 22 intra-frame prediction. In addition, in case when determining that the image is intended for decoding, this is the information interframe coding scheme 41 definitions forecasting 41 outputs the motion vector and flag identification, to the verge stored in the image header, in scheme 21 prediction/motion compensation.

Scheme 41 definitions forecasting additionally transmits the bit stream of the image for which you have defined a method for predicting, in scheme 42 decode processing. Scheme 42 decode processing performs a decoding process corresponding to the encoding scheme, such as the process of decoding variable length or process arithmetic decoding to the image. Scheme 41 determine the forecasting outputs the quantized conversion coefficient, which is obtained by performing the decode processing in scheme 13 dekvantovanie.

In Fig.5 shows a block diagram illustrating an example configuration of a main part schematic prediction/motion compensation Fig.3.

As shown in Fig.5, the circuit 21 prediction/motion compensation includes circuit 51 of the choice of the forecasting scheme 61 unidirectional prediction circuit 62 bidirectional prediction circuit 63 forecasting with increasing frequency and circuit 64 forecasting by filtration. The motion vectors and the flag of identification supplied from the circuit 12 decodes lossless (scheme 41 definition of forecasting), served in scheme 51 the choice of forecasting.

Scheme 51 the choice of forecasting selects a prediction mode in accordance with the flag is automatically recognized, supplied from the circuit 41 to determine the prediction. In case, when determining that the image prediction should be generated using unidirectional prediction circuit 51 of the choice of the forecasting outputs the motion vectors in scheme 61 unidirectional prediction. In addition, in case when determining that the image prediction should be generated through bidirectional prediction circuit 51 of the choice of the forecasting outputs the motion vectors in scheme 62 bidirectional prediction. In addition, in case when determining that the image prediction should be generated by projecting the conversion with increasing frequency, the circuit 51 of the choice of the forecasting outputs its corresponding instruction in scheme 63 forecasting with a transform with increasing frequency.

In addition, in case when determining that the image prediction should be generated as a result of prediction by the filter circuit 51 of the choice of the forecasting outputs the motion vectors in scheme 64 forecasting by filtering.

Thus, to ensure the ability to identify forecasting by filtering, the value differs from the value representing the unidirectional prediction, the value, not only the abuser bidirectional prediction, and the value representing the prediction conversion with increasing frequency, which is defined in the usual standard, can be set as the value of the flag identification.

Scheme 61 unidirectional prediction sets the number of frames arranged in one direction at a time on the extension level, as reference frames, and determines the macroblocks in the reference frames corresponding to the image prediction based on motion vectors. In addition, the circuit 61 unidirectional prediction generates image prediction by reading each of the defined macroblocks in the reference frames from the storage device 19 of the frame as the image motion compensation, and by setting the value of a pixel in one image motion compensation, as the pixel value in the image prediction. Scheme 61 unidirectional prediction outputs the generated image prediction in scheme 15 adder. Examples of unidirectional prediction performed by the circuit 61 unidirectional prediction, include unidirectional prediction, as defined in the standard H.264/SVC (or standard N. 264).

Scheme 62 bidirectional prediction sets the number of frames arranged in time in both directions at the level of RA is spreaders, as keyframes, and determines the macroblock of the reference frame corresponding to the image prediction based on motion vectors. In addition, the circuit 62 bidirectional prediction generates image prediction by reading each of the defined macroblocks in the reference frames from the storage device 19 of the frame as the image motion compensation, and by setting the average of pixel values within the image motion compensation, as the pixel value of the image prediction. Scheme 62 bidirectional prediction outputs the generated image prediction in scheme 15 adder. Examples of bidirectional prediction performed by the circuit 62 bidirectional prediction, include bi-directional prediction, as defined in the standard H.264/SVC (or standard N. 264).

As shown in Fig.1, scheme 63 forecasting with increasing frequency sets the current frame to the Foundation level as the reference frame. Scheme 63 forecasting with increasing frequency selects the macroblock in the same position as the macroblock - purpose processing in the current frame on the extension level of the reference frame on the Foundation level. That is, the circuit 63 forecasting with increasing frequency reads the macroblock of the reference frame on the Foundation level, which corresponds to the acrobacy - purposes of processing from the storage device 19 of the frame. The selected macroblock is a macroblock at the Foundation level, and, therefore, has a lower resolution than the macroblock - purpose processing. Scheme 63 forecasting with increasing frequency converts the selected macroblock-level basis for generating image prediction macroblock - processing purposes.

Scheme 63 forecasting with increasing frequency displays the generated image prediction in scheme 15 adder. Examples of bidirectional prediction performed by the circuit 63 forecasting with increasing frequency, include forecasting with increasing frequency, as defined in the standard H.264/SVC.

As shown in Fig.2, scheme 64 forecasting by filtering defines a set of frames located in one or both directions in time at the Foundation level, as keyframes. Which frames should be used as reference frames may be determined in advance or may be set according to the information transmitted from the encoding together with the flag identification. For example, the previous time frame relative to the current frame and the next preceding frame that is two frames can be set as keyframes. In addition, for example, the previous time frame and following the CR is s relative to the current frame, that is, two frames can be set as keyframes. Of course, other frames can be used as keyframes.

Scheme 64 forecasting by filtering determines the macroblock of the reference frame on the Foundation level, defined as described above, which correspond to the image prediction based on motion vectors supplied from the circuit 51 of the choice of forecasting. Scheme 64 forecasting by filtering reads each set of macroblocks of the reference frame as an image motion compensation, from the storage device 19 of the frame. It should be noted that the motion vectors can be determined instead of units of macroblocks 16x16 pixels and so on, in units of blocks, which are additionally divided macroblock.

Image motion compensation represent the image at the Foundation level, and therefore have a lower resolution than the macroblock - processing purposes, on the extension level. Scheme 64 forecasting by filtration filters, implying a prediction conversion with increasing frequency, using image motion compensation, as an input, and outputs the image prediction obtained by performing the filtering circuit 15 of the adder. The image prediction was transformed with increasing h is the frequency to resolution macroblock-level extensions.

Scheme 64 forecasting by filtering displays the generated image prediction in scheme 15 adder.

In Fig.6 shows a block diagram illustrating an example configuration of the main part of the circuit 64 prediction filter by Fig.5. In scheme 64 forecasting by filtering, with the configuration according to Fig.6, the filtering is done on the signal in the area of time.

As shown in Fig.6, the circuit 64 forecasting by filtering includes 71 circuit selection and circuit 72 of the filter. Scheme 71 selection sets keyframes on the Foundation level, on the basis of information supplied from the circuit 51 selecting forecasting, and allocates the image motion compensation (for example, the image MCO motion compensation and image MC motion compensation from a reference frame at the Foundation level.

One of the suitable tools for the identification of multiple low-resolution images used for the present invention must use signals in the stream at the Foundation level, without summation new signals.

Thus, the method consists in the fact that two images are used as input data for subsequent processing filter, where the first entry represents the decoded image in the same spatial position in the image with low RA is the solution at the current time, and the second input is a time-past or future image with a lower resolution than the image used for prediction in time.

That is, in this case, the circuit 71 selection selects, as one image motion compensation, the macroblock of the current frame on the Foundation level, which is in the same position as the macroblock - purpose processing on the extension level, and allocates another image motion compensation using the motion vector that was used to decode the macroblock at the Foundation level. The advantage of this technology lies in the fact that no new summation of the signals in the stream is not preferred from the point of view of coding efficiency.

At this time, when information related to the set of reference frames used for decoding in the image with a low resolution, in particular, when performing bidirectional prediction, etc., all of the images are forecasting can be used as second and third input data.

Usually, the greater use of information on time with high correlation, the higher the resolution can be generated in the post-processing filtering. Thus, mounted above method is eff is active.

In addition, in order to more accurately generate a high-resolution image during processing, filtering, encoding again one or multiple motion vectors can also be used.

In this case, a new motion vector that is separate from the motion vectors used in decoding the Foundation level, encode to decode the extension level. In this way a new signal is added to the stream; however, the increase in prediction accuracy for level expansion allows to reduce the residual signals at the extension level. Thus, this method can be effective from the point of view of coding efficiency.

Scheme 71 selection sets the image MC motion compensation and image MC motion compensation, as described above, and receives information in respect of them from the storage device 19 of the frame. Scheme 71 selection delivers the selected image MC motion compensation and image MC motion compensation in the circuit 72 of the filter.

Scheme 72 of the filter performs filtering, including forecasting with the conversion frequency of the supplied images MS motion compensation and image MC motion compensation, and generates the image prediction. That is, the circuit 72 of the filter performs processing Phi is Tracii to compensate for a variety of image motion compensation, selected by circuit 71 in relation to the high-frequency components, using a correlation in a time direction that is included in the image motion compensation to generate the image prediction with a higher resolution than the image motion compensation. The image prediction generated thus was compensated for the high-frequency components. Thus, improve the accuracy of prediction. Resulting in improved coding efficiency.

As shown in Fig.6, the circuit 72 of the filter includes a circuit 81 of the difference calculation, the scheme 82 conversion with increasing frequency, the circuit 83 filter of low frequencies, the circuit 84 gain control, the circuit 85 filter high frequency circuit 86 gain control, circuit 87 adder circuit 88 conversion with increasing frequency and circuit 89 adder.

Image MC motion compensation is supplied from the circuit 71 selection in scheme 81 calculation of the difference in scheme 88 conversion with increasing frequency, and the image MC motion compensation is applied to the circuit 81 of the difference calculation.

In the case when the image prediction should be generated through unidirectional prediction, for example, the image extracted from the reference frame R0 next to the current frame, which can be regarded as having more than the high correlation, than the image prediction, is used as the image MC motion compensation, and the image extracted from the reference frame R1, located further away from current frame is used as the image MC motion compensation. The image extracted from the reference frame R0, can be designed for use as an image MC motion compensation, and the image extracted from the reference frame R1, can be designed for use as an image MC motion compensation.

In addition, when the image prediction should be generated through bidirectional prediction, for example, the image is separated from the preceding anchor frame L0, is used as the image MC motion compensation, and an image selected from the following anchor frame L1, is used as the image MC motion compensation. The image extracted from the reference frame L0 may be designed for use as image MC motion compensation, and the image extracted from the reference frame L1, may be designed for use as image MC motion compensation.

Scheme 81 calculating the difference calculates the difference between the image MC motion compensation and image MC motion compensation using, for example, Equation (1), PR is stavlennie below, and outputs the differential image D in scheme 82 conversion with increasing frequency

D(i,j)=A(i,j)-B(i,j)(1)

In Equation (1) (i, j) represents the relative position of a pixel in the image motion compensation. For example, in the case when the process is submitted for execution in units of macroblocks of 16×16 pixels, set 0≤i≤16 and 0≤j≤16. This similarly applies to the following description.

Scheme 82 conversion with increasing frequency performs the conversion resolution differential image D, calculated in the circuit 81 of the difference calculation. The ratio of the conversion resolution is based on the ratio of the spatial resolution for the Foundation level to the level of expansion. For example, when the resolution for the Foundation level is n×m [pixels] (n and m are integers), and for the extension level is N×M [pixels] (N and M are integers, where N>n, and M>m), the magnification in the horizontal direction H_Scale and the magnification in the vertical direction V_Scale will be given by Equation (2) and Equation (3)

H_Scale=N/n (2)

V_Scale=M/m(3)

Scheme 82 conversion with increasing frequency outputs the differential image D', which has been subjected to resolution conversion (was transformed with increasing frequency), thus, in scheme 83 filter low frequency.

Scheme 83 filter low frequency includes a diagram of the FIR filter. In scheme 83 filter low frequency apply the filter of low frequencies to the differential image D' supplied from the circuit 82 conversion with increasing frequency, and display the resulting image in scheme 84 gain control and circuit 85 filter high frequency. The differential image D", that is, the image obtained by applying the filter of low frequencies represent using Equation (4) as follows:

D=LPF (D') (4)

In Equation (4) LPF (X) represents the application of the filter of low frequencies to the input image X using a two-dimensional FIR filter.

Scheme 84 gain control adjusts the gain of the differential image D supplied from the circuit 83 filter of low frequencies, and displays the image with the adjusted gain circuit 87 adder. If set 0≤I≤16×H_Scale and if set 0≤J≤16×V_Scale, the output image X (I, J) scheme 84 gain control will be represented by the Equation (5) as follows:

X(I,J)=αD(I,J)(5)

Scheme 85 high-pass filter includes a diagram of the FIR filter. Scheme 85 high-pass filter applies a high-pass filter for differential image D supplied from the circuit 83 filter low frequency, and outputs the resulting image in scheme 86 gain control. The differential image D', that is, the image obtained by applying a filter the high frequency represented by Equation (6) as follows:

D"'=HPF (D)(6)

In Equation (6) HPF(X) represents the operating characteristics of the process of filtering high frequency in the input image X using a two-dimensional FIR filter.

Scheme 86 gain control adjusts the gain of the differential image D' supplied from the circuit 85 filter high frequency, and displays the image with the adjusted gain circuit 87 adder. The output image Y (I, J) circuit 86 gain control is represented by the Equation (7) as follows:

Y(U)=βD'(I,J)(7)

As the value of α in Equation (5) and the values of β in Equation (7), for example, choose values such as α=0.8 and β=0.2 is. However, other values can be used to increase the accuracy of pixel prediction. In addition, these values can be adaptively changed in accordance with characteristics of the input sequence, etc.

Scheme 87 adder sums the image is (I, J) with the adjusted gain and the image Y (I, J), and displays the image obtained by summation. The output image Z(I, J) scheme 87 adder is represented by Equation (8) as follows:

Z(I,J)=X(I,J)+Y(I,J)(8)

The output image Z (I, J) is the representation of high-frequency components of the image, which may be defined differently, i.e. the correlation between the images MS motion compensation and image MC motion compensation.

Scheme 88 conversion with increasing frequency performs resolution conversion of the image MC motion compensation. As in the case of circuit 82 conversion with increasing frequency, the ratio of the resolution conversion based on the ratio of the spatial resolution for the Foundation level to the level of expansion. That is, the magnification in the horizontal direction H_Scale and the magnification in the vertical direction V_Scale given by Equation (2) and Equation (3) above. Scheme 88 conversion with increasing frequency gives the image A', which represents the image MC motion compensation, vergote the resolution conversion (converted with increasing frequency), as described above, the circuit adder 89.

Scheme 89 adder sums the output image Z (I, J) supplied from the circuit 87 adder with the image And' supplied from the circuit 88 conversion with increasing frequency, and outputs the resulting image in scheme 15 adder, as picture prediction. The image S (I, J) forecasting, which ultimately derive from the scheme 89 adder presented in Equation (9) as follows:

S(I,J)=A'(I,J)+Z(I,J)(9)

Thus, in accordance with the mode of prediction by filtering the image that represents the high-frequency components and which is generated by transformation with increasing frequency image at the Foundation level, add to the image obtained by the prediction conversion with increasing frequency image MC motion compensation. Thus, generate the image prediction.

In the generate image prediction in the prediction mode by filtering, as described above, the decoding device 1 can receive image prediction, glucouse more high-frequency components, than the image prediction when the prediction conversion with increasing frequency, which is obtained in the result of the conversion with increasing frequency image of the current frame on the Foundation level. In addition, since the filtering is performed in such a manner as described above, when the image prediction generated from a variety of image motion compensation, the decoding device 1 can receive image prediction that includes more high-frequency components than the image prediction, which has just as each pixel value, the average value of corresponding pixels of the set of image motion compensation.

In addition, the resolution of the image to which you want to apply, less than that of the image intended for interframe prediction, in which the image prediction generated by recourse to human resources at extension level. In line with this, there is no need to save images in high resolution on the extension level in the storage device 19 of the frame or read from the storage device 19 of the frame. In addition, for example, as the motion vectors, information during decoding of the Foundation level can be used during decoding of the extension level. Thus the set reduced the amount of code for the information of the compressed image. That is, the decoding device 1 can improve the compression efficiency.

Thus, the decoding device 1 can improve the coding efficiency, excluding the increase in load.

Explanations stream decode processing

Next, explain the processing performed by the decoding device 1 having the above configuration. First, explained is an example of the flow of the decoding process for the extension level with reference to the block diagram of the sequence of operations shown in Fig.7. As in the case of the decoding process for Foundation level, decoding for extension level also mainly performed using a method that meets the standard N. 264.

In this respect, the decoding process for the level of expansion substantially different from that of the decoding process for Foundation level or standard N. 264, in which there is a mode in which the image on the Foundation level is simultaneously used to generate the image prediction. In addition, in the case of the decoding process for the extension level which is applied the present invention add the function to use multiple images at the Foundation level, which at the time are in the same time or at different points in time relative to the current frame, on which I decode the extension level.

The process according to Fig.9 starts when, for example, an image with a specific size, such as a macroblock 16x16 pixels, read with schema 12 decoding without loss of the information stored in the buffer 11 save. The processing of each step in Fig.9 is accordingly, in parallel with the processing of another phase or as a result of changing the order of execution stages. It's the same extent relates to the handling of each stage in each block diagram of the sequence of operations described below.

At step S1, the circuit 12 decodes lossless begins the process of decoding lossless image read from the buffer 11 save. Details of the decoding process without loss will be described below. Scheme 12 decode lossless outputs the quantized conversion coefficient generated by using the decoding process without losses in scheme 13 elimination of quantization. In addition, the circuit 12 decodes lossless displays information about the mode intraframe prediction circuit 22 intraframe prediction 22 when the image intended for decoding, is an image intraframe coding, and outputs the motion vectors and the flag identification scheme 21 prediction/motion compensation when the picture is of is an image, encoded between frames in the decoding process without losses.

At step S2 figure 13 removing quantization performs elimination of quantization using the schema corresponding to the quantization scheme used on the side coding, and outputs the conversion coefficient in scheme 14 inverse orthogonal transformation. At step S3 scheme 14 inverse orthogonal transformation performs inverse orthogonal conversion by the conversion factor originating from circuit 13 eliminate quantization, and outputs the resulting image in scheme 15 adder.

At step S4 scheme 15 adder combines the decoded image filed from the circuit 14 inverse orthogonal transform, and the image prediction filed from the circuit 21 prediction/motion compensation, or circuit 22 intra-frame prediction, and outputs the composite image to the filter 16 address blocks. At step S5, the filter 16 address blocks performs filtering to remove noise block included in the composite image, and displays the image, from which you have removed the noise blocks. At step S6 storage device 19 of the frame temporarily stores the image submitted from the filter 16 address blocks. In addition, at this time, the image is also retained in the buffer 17 changes the layout.

On this the e S7 scheme 31 management determines was or not the above process is executed for the macroblocks in one entire frame. In case, when determining that the process was not completed, attention is focused on another macroblock and repeatedly performs the process from step S1.

In addition, when in step S7 determines that the process was performed on the macroblocks in the entire frame, the process moves to step S8. At step S8, the buffer 17 changes the layout displays the generated frame to the circuit 18 D/A Converter in accordance with the control circuit 31 of the control.

At step S9 scheme 18 D/A Converter performs D/A conversion for frame filed from the buffer 17 changes the layout, and outputs the analog signal to the outside. The above-described processing is performed for each frame.

Next, with reference to the block diagram of the sequence of operations shown in Fig.8 will be described an example of a processing flow of the decoding lossless.

When it starts the decoding process without losses, at step S21, the circuit 41 definitions forecasting refers to the header information of the compressed image submitted from the buffer 11 save. At step S22 scheme 41 definitions forecasting determines whether to perform or not intra-frame prediction based on the information indicating the prediction mode set by the encoding device, which is included in sagola is OK. When the mode is intra-frame prediction was installed using the encoder, the process goes to step S23.

At step S23 scheme 22 intraframe prediction performs intra-frame prediction to generate image prediction, and gives an image prediction in scheme 15 adder. The image prediction is combined with the decoded image supplied from the circuit 14 of the inverse orthogonal transform in step S4 in Fig.7.

When the processing at step S23 is completed, the processing goes to step S29. In addition, if at the step S22 determines that the intra-frame prediction should not be performed, the processing goes to step S24.

At step S24 scheme 41 definitions forecasting determines whether or not to perform the prediction conversion with increasing frequency on the basis of information indicating the prediction mode set by the encoding device, which is included in the header. In the case where the prediction mode with a conversion frequency was set by the encoding device, the processing goes to step S25.

At step S25 scheme 63 predictive transform with raising frequency circuit 21 prediction/motion compensation performs the prediction conversion with increasing h is the frequency, to generate the image prediction, and supplies this image prediction in scheme 15 adder. The image prediction is combined with the decoded image supplied from the circuit 14 inverse orthogonal transformation on the step S4 in Fig.7.

When the processing at step S25 ends, the processing proceeds to step S29. In addition, when in step S24 determines that the prediction conversion with increasing frequency should not be executed, the processing goes to step S26.

At step S26 scheme 41 definitions forecasting determines should be made interframe prediction on the basis of information indicating the prediction mode determined by the encoding device, included in the header. When the mode interframe prediction was determined by the encoding device, the processing goes to step S27.

At step S27 circuit 61 unidirectional prediction or circuit 62 bidirectional prediction from the circuit 21 prediction/motion compensation performs interframe prediction (unidirectional prediction or bidirectional prediction), to generate the image prediction, and gives an image prediction in scheme 15 adder. The image prediction combined with the decoded what images, served from circuit 14 inverse orthogonal transformation on the step S4 in Fig.7.

When the processing in step S27 is ended, the processing goes to step S29. In addition, when in step S26 determines that the prediction mode by filtration was determined by the encoding device and that interframe prediction should not be executed, the processing proceeds to step S28.

At step S28 scheme 64 forecasting by filtering scheme 21 prediction/motion compensation performs prediction filter to generate image prediction based on the information indicating the prediction mode by filtering, which is included in the header, and delivers the image prediction in scheme 15 adder. The image prediction is combined with the decoded image supplied from the circuit 14 of the inverse orthogonal transform in step S4 in Fig.7. When the processing in step S28 is completed, the processing goes to step S29.

At step S29 scheme 42 processing of decoding decodes the residual signal of the compressed image information and outputs the quantized coefficient conversion circuit 13 dekvantovanie. When the processing at step S29 is finished, the processing of decoding lossless ends. Then, the processing returns to step S1 in Fig.7 and executes processing is and after step S2.

It should be noted that the above explained that the prediction mode is chosen on the basis of information included in the header of the compressed image information that is accessed at step S21. However, this is not a limitation, and the circuit 41 define a prediction can be made with the possibility of selecting an appropriate prediction mode, for example, by analyzing the bit stream information of the compressed image. In this case, at step S21 instead of going to the header circuit 41 definitions forecasting analyzes the information of the compressed image, and selects the prediction mode on the basis of the analysis in the processing in step S22 and subsequent steps.

Next, an example of a processing flow forecasting by filtering performed in the processing at step S28 in Fig.8, is illustrated with reference to the block diagram of the sequence of operations shown in Fig.9.

When processing begins prediction by filtering, at step S41, the circuit 71 allocation allocates the image motion compensation of the current frame or reference frames from the Foundation level. At step S42 scheme 81 calculating the difference calculates the difference between the image motion compensation. At step S43 scheme 82 conversion with increasing frequency transforms with increasing frequency difference between the images of the compensation movement is Oia, calculated in step S42. At step S44 filter circuit 83 low frequency applies the filter to the low frequency to the difference obtained after conversion with increasing frequency at step S43.

At step S45 scheme 84 gain control multiplies the output of the filter a low frequency when the processing in step S44 by the coefficient α to perform gain control. At step S46 scheme 85 high-pass filter applies a high-pass filter to the output of the filter of low frequencies in the processing at step S44. At step S47 scheme 86 gain control multiplies the output of the filter high frequency when the processing at step S46 by a factor β to perform gain control.

At step S48 scheme 87 adder sums the output with the adjusted gain of the filter of low frequencies in the processing at step S45 and the output with the adjusted gain of the high-pass filter in the processing at the step S47 to determine the components of the high frequency.

At step S49 scheme 88 conversion with increasing frequency performs the conversion with increasing frequency image MC motion compensation allocated from the Foundation level. At step S50 scheme 89 adder adds high frequency components, as identified in step S48, the image motion compensation obtained after conversion with increasing frequency in the step S49, to generate the image the Oia forecasting. Scheme 89 adder supplies the generated image prediction in scheme 15 adder.

When the processing at step S50 is completed, the processing prediction by filtering ends. The process then returns to step S28 in Fig.8, and executes the processing after step S29.

As noted above, the decoding performed using the image prediction generated during the prediction by filtering, thus, allows to obtain a decoded image with high resolution, without increasing the load on the processing. That is, the decoding device 1 can improve the coding efficiency by preventing the increase of the load.

It should be noted that the above explained that the decoding of the Foundation level and the decoding of the extension level is performed by using the same device 1 decoding. However, this is not a limitation, and decoding at both levels can be performed using different devices 1 decoding. In this respect, also in this case, the storage device 19 of the frame is common to all devices, decoding, and frame on the Foundation level can be performed which can be read during decoding of the extension level.

2. The second option exercise

Device configuration codiovan the

In Fig.10 shows a block diagram illustrating an example configuration of the main part of the encoder that uses the present invention. The device 101 encoding is an encoding device corresponding to the decoding device 1 in Fig.3. That is, the compressed image information obtained by performing the encoding device 101 coding, introduced in the decoding device 1 according to Fig.3.

The device 101 encoding includes circuit 111 A/D Converter, the buffer 112 composition changes, the circuit 113 adder circuit 114 orthogonal conversion circuit 115 quantization circuit 116 lossless encoding and buffer 117 save. The device 101 encoding additionally includes circuit 118 speed control, circuit 119 dekvantovanie, the circuit 120 inverse orthogonal transform, filter 121 removing the blocks, the memory device 122 of the frame and circuit 123 definition mode. In addition, the device 101 encoding includes a switch 124, the circuit 125 prediction/motion compensation circuit 126 intra-frame prediction, the switch 127 and circuit 131 controls.

Image information is divided into two levels (or on many levels more than two levels, i.e. the level of the base of the low resolution and the extension level with high resolution is receiving, and image information for each frame on the Foundation level with a low resolution pre-served in the device 101 encoding and encode. Encoding Foundation level perform similarly in accordance with the standard N. 264. When the encoding of the Foundation level is completed, the image information on the extension level code using device 101 encoding. The encoding of the extension level is illustrated below.

Scheme 111 A/D Converter performs A/D conversion of the input signal and displays the image in the buffer 112 composition changes. The buffer 112 composition changes personnel changes in accordance with the structure of a GOP (group of pictures) of the compressed image information and displays the image of a particular module, such as a macroblock. The image output from the buffer 112 composition changes, is fed into the circuit 113 adder circuit 123 definition mode, the circuit 125 prediction/motion compensation and circuit 126 intra-frame prediction.

Scheme 113 adder determines the difference between the image supplied from the buffer 112 composition changes, and the image prediction generated by the scheme 125 prediction/motion compensation, or circuit 126 intra-frame prediction, and supplied through the switch 127, and outputs the remainder of the circuit 114 orthogonal transformations. Than b the who picture prediction to the original image and the smaller the number of remainders defined here, the less code you want to assign to the residue, and, therefore, we can say, the higher the coding efficiency.

Scheme 114 orthogonal transformation performs orthogonal transform such as discrete cosine transformation or conversion of karunen-Loev for the remainder supplied from the circuit 113 of the adder, and outputs the conversion coefficient obtained by performing orthogonal transformation circuit 115 quantization.

Circuit 115 performs quantization quantization conversion factor supplied from the circuit 114 orthogonal transformation in accordance with the control by the circuit 118 speed control, and outputs the quantized conversion coefficient. The conversion coefficient is quantized by using the schema 115 quantization, is fed into the circuit 116 lossless encoding and circuit 119 dekvantovanie.

Scheme 116 lossless encoding compresses the conversion factor supplied from circuit 115 quantization, by performing lossless encoding such as variable length coding or arithmetic coding, and outputs data in the buffer 117 is saved.

In addition, the circuit 116 lossless encoding sets the value of the flag identification in accordance with the information supplied from the circuit 123 definition wide-angle the mode, and describes this flag identification in the image header. As described above, the decoding device 1 determines the prediction mode on the basis of the flag identification described circuit 116 lossless encoding.

Scheme 116 lossless encoding also describes the information supplied from the circuit 125 prediction/motion compensation or circuit 126 intra-frame prediction, in the image header. The motion vectors, etc., that detects when performing inter-frame prediction, is passed from the circuit 125 prediction/motion compensation, and information related to the intra-frame mode prediction, served from circuit 126 intra-frame prediction.

The buffer 117 save temporarily stores information supplied from the circuit 116 lossless encoding, and outputs it as a compressed image information at a certain point in time. The buffer 117 saving displays information about the amount of generated code in the schema 118 speed control.

Scheme 118 speed control calculates the quantization scale based on the amount of code that is output from the buffer 117 conservation, and manages the scheme 115 quantization in such a way that the quantization can be performed with the calculated quantization scale.

Scheme 119 dekvantovanie performs dekvantovanie on the calculated coefficients is the conversion quantized using schema 115 quantization, and outputs the coefficient conversion circuit 120 inverse orthogonal transformation.

Scheme 120 inverse orthogonal transform performs inverse orthogonal transform on the coefficient of conversion, is transferred from circuit 119 dekvantovanie, and displays the resulting image in the filter 121 removing the blocks.

The filter 121 removing blocks removes noise blocks that occur in the locally decoded image, and outputs an image from which you have removed the noise blocks in the storage device 122 of the frame.

In the storage device 122 of the frame contains the image transferred from the filter 121 removing the blocks. The image stored in the storage device 122 of the frame, read with schema 123 definition mode accordingly.

Scheme 123 definition mode determines whether to perform intra-frame coding or must be performed interframe coding, on the basis of the image stored in the storage device 122 of the frame, and the image transferred from the buffer 112 composition changes. In addition, in case when determining what should be done interframe coding scheme 123 definition mode defines one mode among the modes of the unidirectional prediction mode is and bidirectional prediction, the prediction mode conversion with increasing frequency and mode prediction by filtration. Scheme 123 definition mode displays information denoting the result of the determination, the circuit 116 lossless encoding as information mode.

When determine what must be done interframe coding scheme 123 definition mode displays frames, which are stored in the storage device 122 of the frame and which are the result of local decoding, the circuit 125 prediction/motion compensation through the switch 124.

In addition, when determine what must be performed intra-frame coding circuit 123 definition mode displays frames, which are stored in the storage device 122 of the frame and which are the result of local decoding, the circuit 126 intra-frame prediction.

The switch 124 is connected to the output a11 when you want to perform interframe encoding, and it is connected to the output b11 when should be performed intra-frame coding. Toggle switch 124 is controlled, for example, using the circuit 131 controls.

Scheme 125 prediction/motion compensation detects the motion vectors on the basis of the original image supplied from the buffer 112 composition changes and supporting the frame, read from the storage device 122 of the frame, and outputs the detected motion vectors to the circuit 116 lossless encoding. In addition, the circuit 125 prediction/motion compensation generates image prediction by performing motion compensation using the detected motion vectors and reference frames, and outputs the generated image prediction scheme adder 113 through the switch 127.

Scheme 126 intra-frame prediction performs intra-frame prediction on the basis of the original image transmitted from the buffer 112 composition changes, and the reference frame, the locally decoded and stored in the storage device 122 of the frame, and generates the image prediction. Scheme 126 intra-frame prediction outputs the generated image prediction scheme adder 113 through the switch 127 and displays information about the mode intraframe prediction circuit 116 lossless encoding.

The switch 127 is connected with the output A12 or output b12, and it displays the image prediction generated by the scheme 125 prediction/motion compensation or circuit 126 intra-frame prediction, the circuit 113 adder.

Circuit 131 controls the overall operation of the device 101 encoding, for example, by switching from the organisations switches 124 and 127 in accordance with the regime a specific circuit 123 definition mode.

In Fig.11 shows a block diagram illustrating an example configuration of a main part schematic 123 determination mode according to Fig.10.

As shown in Fig.11, the circuit 123 definition mode includes circuit 141 intraframe prediction circuit 142 unidirectional prediction circuit 143 bidirectional prediction circuit 144 forecasting with a transform with increasing frequency, the circuit 145 forecasting by filtering, circuit 146 calculation of the prediction error and circuit 147 definition.

In the scheme 123 determine the mode of each of the intra-frame prediction and interframe prediction performed for a block having a different size, and determine a prediction mode should be performed on the results. As for interframe prediction, the processing performed in each of the prediction modes, that is, in the mode unidirectional prediction, bidirectional prediction, the prediction mode conversion with increasing frequency and in prediction mode by filtering.

Scheme 141 intraframe prediction circuit 142 unidirectional prediction circuit 143 bidirectional prediction circuit 144 forecasting with a transform with increasing frequency and circuit 145 prognose the Finance by filtering makes predictions using separate methods for generating image prediction based on the original image and the image read from the storage device 122 of the frame, and output the generated image prediction in the scheme 146 calculate the prediction error.

Scheme 141 intra-frame prediction performs intra-frame prediction by using a method similar to the method used in figure 22 intraframe prediction decoding device 1. The circuit 142 unidirectional prediction detects the motion vectors, selects image motion compensation from a reference frame based on the detected motion vectors and performs unidirectional prediction using image motion compensation to generate the image prediction. That is, the circuit 142 unidirectional prediction generates image prediction using a method similar to the method of scheme 61 unidirectional prediction decoding device 1 on the basis of the detected motion vectors.

Scheme 143 bidirectional prediction detects the motion vectors, selects image motion compensation from a reference frame based on the detected motion vectors and performs bidirectional prediction image using the motion compensation, to generate the image prediction. That is, the scheme 143 bidirectional prediction generates image prediction using a method similar to that performed by the circuit 62 bidirectional prediction decoding device 1 on the basis of the detected motion vectors.

The circuit 144 forecasting with a transform with increasing frequency sets the macroblock of the current frame on the Foundation level, which is in the same position as the target macroblock processing of the current frame on the extension level, as the image motion compensation, and performs the conversion with increasing frequency image motion compensation to generate the image prediction on the extension level. That is, the prediction scheme 144 conversion with increasing frequency generates the image prediction using method similar to those used in scheme 63 forecasting with the conversion frequency of the decoding device 1.

Scheme 145 forecasting by filtering detects the motion vectors at the Foundation level, allocates image motion compensation on the Foundation level of the reference frames based on the detected motion vectors and performs prediction filtering using image motion compensation at the level of the core is of, to generate the image prediction. That is, the circuit 145 forecasting by filtering generates the image prediction using the method similar scheme 145 forecasting by filtering device 1 decoding on the basis of the detected motion vectors.

It should be noted that all schemas from the schema 141 intra-frame prediction scheme to 145 forecasting by filtering detects the motion vectors or performing prediction in units of, for example, blocks of 4 × 4 pixel blocks of 8x8 pixels blocks and 16×16 pixels. The block size that is used as a processing extension is arbitrary. In addition, the number of blocks for which must be fulfilled prediction, is also arbitrary. All schemas from the schema 141 intra-frame prediction scheme to 145 forecasting by filtering generate image prediction for each block and output the generated individual images forecasting scheme 146 calculate the prediction error.

The original image supplied from the buffer 112 composition changes, enters the schema from the schema 141 intra-frame prediction scheme to 145 forecasting by filtering and circuit 146 calculate the prediction error.

The circuit 146 calculate the prediction error determines the differential is between each of the images forecasting, transferred from the respective circuits in the circuit 141 intra-frame prediction and the original image, and outputs a residual signal representing a certain difference in the scheme 147 definitions. Similarly, the circuit 146 calculate the prediction error determines the difference between each of the images forecasting transferred from the circuit 142 unidirectional prediction circuit 143 bidirectional prediction circuit 144 forecasting with a transform with increasing frequency and circuit 145 forecasting by filtration and the original image, and outputs a residual signal representing a certain difference in the scheme 147 definition.

Scheme 147 definition measures the intensity of the residual signals from the circuit 146 calculation of the forecast error, and determines, as a method for predicting, for generating image prediction, which is used for encoding, a method for predicting, which was used to generate the image prediction with a small difference with the original image. Scheme 147 definition displays information representing the result of the determination, the circuit 116 lossless encoding as information about the mode. Mode information includes information representing the block size that is intended for the use as a processing unit, and so on

In addition, in case when determining that the image prediction should be generated through interframe prediction (in the case, when determining that the interframe coding is to be performed), the scheme 147 definition displays keyframes read from the storage device 122 of the frame, together with information on the mode, the circuit 125 prediction/motion compensation. In case, when determining that the image prediction should be generated through intra-frame prediction (when determine what must be performed intra-frame coding), schema, 147 definition displays the image to be used for intra-frame prediction, which is read from the storage device 122 of the frame, the circuit 126 intra-frame prediction, together with information about the mode.

In Fig.12 shows a block diagram illustrating an example configuration of a main part schematic 125 prediction/motion compensation according to Fig.10.

As shown in Fig.12, the circuit 125 prediction/motion compensation includes circuit 151 detection of the motion vector, the circuit 152 unidirectional prediction circuit 153 bidirectional prediction circuit 154 prediction conversion with increasing frequency and circuit 155 filtering. Scheme 125 forecasting/compens what the movement has a similar configuration with the circuit 21 prediction/motion compensation, presented on Fig.5, except that the circuit 151 detection of the motion vector is provided instead of the circuit 51 selecting forecasting.

Scheme 151 detection of the motion vector detects motion vectors by performing the mapping blocks, etc. on the basis of the original image supplied from the buffer 112 composition changes and reference frames transmitted from the circuit 123 definition mode. Scheme 151 detection of the motion vector refers to information regime transmitted from the circuit 123 definition mode, and displays the keyframes in one of the circuits from the circuit 152 unidirectional prediction scheme to 155 forecasting by filtration. In addition, scheme 151 detection of the motion vector is also outputs the motion vectors at the destination, which must be displayed keyframes as needed.

Scheme 151 detection of the motion vector outputs the motion vectors, together with keyframes, the circuit 152 unidirectional prediction, when it was selected unidirectional prediction, and outputs these pieces of information in the schema 153 bidirectional prediction, when it was selected to perform bi-directional prediction. In addition, when the selected prediction conversion with increasing frequency, scheme 151 detective the Oia motion vector displays an image of the current frame on the Foundation level, which is a reference frame, the circuit 154 prediction conversion with increasing frequency. In addition, when the selected prediction filtering, scheme 151 detection of the motion vector outputs the motion vectors, together with keyframes on the Foundation level, the circuit 155 forecasting by filtering.

The circuit 152 unidirectional prediction generates image prediction means, similar to the scheme 61 unidirectional prediction in Fig.5, execution of the unidirectional prediction. The circuit 152 unidirectional prediction outputs the generated image to the prediction circuit 113 adder. Scheme 153 bidirectional prediction generates image prediction means, similar to the scheme 62 bidirectional prediction in Fig.5, perform bi-directional prediction. Scheme 153 bidirectional prediction outputs the generated image to the prediction circuit 113 adder. The circuit 154 prediction conversion with increasing frequency generates the image prediction means, similar to the scheme 63 forecasting with a transform with increasing frequency in Fig.5, predict conversion with increasing frequency. The circuit 154 prediction conversion with increasing h is the frequency displays the generated image prediction circuit 113 adder.

Scheme 155 forecasting by filtering generates the image prediction means, similar to the scheme 64 prediction filter by Fig.5, selection of image motion compensation of each of the multiple reference frames at the Foundation level and predict by filtering, which includes the conversion frequency sets the selected image motion compensation. Scheme 155 forecasting by filtering displays the generated image prediction circuit 113 adder. It should be noted that the circuit 155 forecasting by filtering has the configuration similar to the configuration schema 64 forecasting by filtering, is shown in Fig.6. Below the circuit configuration 155 forecasting by filtering explained with reference to the circuit configuration 64 forecasting by filtering shown in Fig.6, respectively.

The image prediction generated during the prediction by filtering, can be an image that includes more high-frequency components than the image prediction generated through unidirectional prediction, bidirectional prediction or forecasting with a transform with increasing frequency, and which has a little different is I from the original image. Therefore, the prediction filter requires only a small amount of code assigned to a residue, which improves the coding efficiency. In addition, when the prediction filter resolution reference frames will be lower than in the case of unidirectional prediction or bidirectional prediction, which perform the conversion rate of expansion, resulting in a get a small load when performing processes such as preservation of key frames in the storage device 122 of the frame and the reading of the reference frames from the storage device 122 of the frame. That is, when using prediction by filtering device 101 encoding can improve the coding efficiency, preventing an increase in load during encoding or decoding.

In addition, the prediction filtering may be performed using at least two reference frames. Thus, this increase coding efficiency becomes feasible without increasing the complexity of the processing. The remainder of the original image can be reduced, and the coding efficiency can be increased, for example, by increasing the number of reference frames, intended to be used for interframe prediction, to generate image PR is generowania with high accuracy when using it. However, in this case, the number of reference frames increases, and increases the complexity of processing.

It should be noted that, when choosing a method for predicting, may be the best way to predict, given the number code for information such as motion vectors required for prediction, and the encoding mode, and by adding a weight corresponding to the amount of code to the intensity of the residual signal. This can further improve the coding efficiency. In addition, for a simplified process of encoding method of forecasting can be adaptively selected using feature values of the input original image in the spatial and temporal directions.

The explanation of the flow encoding

Next is illustrated the processing device 101 encoding having the above described configuration.

The encoding process for the extension level, which is performed using the device 101 encoding is explained with reference to the block diagram of the sequence of operations shown in Fig.13. This process begins when the image of a particular module, such as the macroblock output from the buffer 112 composition changes. It should be noted that, as described above, the encoding process for Foundation level similar way to the Snov requirements H.264, and his explanation here, therefore, excluded.

At step S101 circuit 113 adder determines the difference between the image supplied from the buffer 112 composition changes, and the image prediction generated by the scheme 125 prediction/motion compensation, or circuit 126 intra-frame prediction, and outputs the remainder of the circuit 114 orthogonal transformations.

At step S102 scheme orthogonal 114 conversion performs orthogonal transformation of the remainder supplied from the circuit 113 of the adder, and outputs the coefficient conversion circuit 115 quantization.

At step S103 circuit 115 quantization quantum the conversion factor supplied from the circuit 114 orthogonal transformations, and outputs the quantized conversion coefficient.

At step S104 scheme 119 dekvantovanie performs dekvantovanie of the conversion coefficient, the quantized scheme 115 quantization, and outputs the coefficient conversion circuit 120 inverse orthogonal transformation.

At step S105 circuit 120 inverse orthogonal transform performs inverse orthogonal conversion by the conversion factor transmitted from the circuit 119 dekvantovanie, and displays the resulting image in the filter 121 removing the blocks.

At step S106, the filter 121 removing blocks removes noise blocks, filtering, and outputs the image from which you have removed the noise blocks in the storage device 122 of the frame.

At step S107 in the storage device 122 of the frame retain the image is transferred from the filter 121 removing the blocks.

At step S108 scheme 123 definition mode performs the process of determining mode. In the process of determining modes determine which of the forecasting process must be generated by the image prediction. The details of the process definition mode will be described below.

At step S109 scheme 125 prediction/motion compensation or circuit 126 intra-frame prediction to generate the image prediction mode determined at step S108. This image prediction is used in the processing at step S101.

At step S110 circuit 116 lossless encoding compresses the conversion factor transmitted from the circuit 115 quantization, and outputs the compressed conversion coefficient in the buffer 117 save. In addition, the circuit 116 lossless encoding describes the flag identification in the image header or describes the motion vectors transmitted from the circuit 125 prediction/motion compensation in the image header, in accordance with the information supplied from the circuit 123 definition mode.

At step S111, the buffer 117 save temporarily stores information supplied from the circuit 116 codiovan is lossless.

At step S112 (scheme 31 management determines was or not the above process is executed for the macroblocks in one entire frame. In case, when determining that the process was not completed, attention is focused on another macroblock, and the process is repeatedly performed from step S101.

In contrast, in the case where at step S112 (determine that the process was performed for macroblocks in one entire frame, then, at step S113, the buffer 117 save outputs the compressed image information in accordance with control performed by the circuit 131 controls. The above process is performed for each frame.

Next, the process mode determination performed at step S108 in Fig.13, illustrated with reference to the block diagram of the sequence of operations in Fig.14.

At step S131, each of the schemas from the schema 141 intra-frame prediction scheme to 145 forecasting by filtering performs intra-frame prediction or inter-frame prediction block having different sizes, and generates the image prediction. The generated image prediction is fed to the circuit 146 calculate the prediction error.

At step S132 circuit 146 calculate the prediction error determines the difference between the original image and each of the images forecasting, submitted from the scheme 141 intra-frame prediction in scheme 15 forecasting by filtration. The circuit 146 calculate the prediction error displays the residual signals in the circuit 147 definition.

At step S133 scheme 147 definition specifies a method for predicting, for generating image prediction intended for filing in the circuit 113 adder based on the intensity of the residual signals from the circuit 146 calculate the prediction error.

On the stage set s134 scheme 147 definition displays information about the mode, which is information relating to a specific method of forecasting, the circuit 116 lossless encoding. After that, the processing returns to step S108 in Fig.13, and executes the subsequent processing.

Next, an example of a process flow forecasting by filtering, to generate image prediction by performing prediction by filtration, is explained with reference to the block diagram of the sequence of operations in Fig.15, as an example of a process for generating image prediction performed at step S109 in Fig.13.

As described above, at step S109 of Fig.13, the image prediction generated using the mode specified in the process definition mode, at step S108. In accordance with this, in the case where the prediction mode by filtering determines at step S108, step S109 to perform the forecasting process by Phi is Tracie, as is shown in Fig.15.

When you start the forecasting process by filtering, at step S151, scheme 151 detection of the motion vector detects the motion vectors on the basis of the original image and reference frames.

When detects motion vectors, perform the processing at steps S152 - S161, using the detected motion vectors, similarly to the processing performed in steps S41-S50 of Fig.9, respectively. That is, image motion compensation generate keyframes on the Foundation level, based on the motion vectors, and the filtering processing, including conversion with increasing frequency, to perform image motion compensation to generate the image prediction at the level of extensions.

When the processing in step S161 is finished, the processing prediction by filtering ends. Then, the processing returns to step S109 in Fig.13, and executes the processing after step S110.

It should be noted that when different modes is selected when the processing at step S108, the circuit 125 prediction/motion compensation or circuit 126 intra-frame prediction to generate the image prediction using the selected another mode. The above-described processing is performed in accordance with standard H.264/SVC, and its explanation here, therefore, excluded.

As noted above, coding, which take into account the spatial scalability is accomplished by more efficient use of temporal correlation is included in the signal sequence in a moving image, providing, thus, for example, improve coding efficiency, while preventing the increase of load processes, such as encoding and decoding.

3. A third option exercise

Overview of the decoding process

In Fig.16 shows a diagram illustrating another example of the overview of the decoding process, which is applied the present invention. As shown in Fig.16, the number of reference frames can be three or more.

In the example shown in Fig.16, temporarily preceding frame relative to the current frame and the next preceding frame, and still further preceding frame which is three frames (Ref0, Ref1, Ref2) are set as keyframes. Preceding frame relative to the current frame is set as the reference frame Ref0 preceding frame relative to the reference frame Ref0 is selected as the reference picture Ref1, and the preceding frame relative to the reference frame Ref1 is selected as the reference picture Ref2.

The circuit configuration of the filter

In Fig.17 shows a block diagram illustrating an example circuit configuration of the filter according to Fig.6 in this case, when accessing the t is eating human resources.

As shown in Fig.17, the circuit 211 filtering includes circuit 221 filtering and circuit 222 of the filter. Each circuit 221 filtering and circuit 222 filter has a configuration such as shown in Fig.6. Thus, the circuit 211 filtering is executed with the ability to work as a circuit with one output and the circuit with three inputs by cascading circuit 72 of the filter used for design with one output and two inputs.

It should be noted that at this time the circuit 71 allocation allocates the image motion compensation of each of the three reference frames (Ref0, Ref1, Ref2). That is, for example, scheme 71 allocation allocates the image MC motion compensation from a reference frame Ref0. In addition, for example, scheme 71 allocation allocates the image MC motion compensation from a reference frame Ref1. In addition, for example, scheme 71 allocation allocates the image MC2 motion compensation from a reference frame Ref2.

Image MC and MC2 motion compensation is applied to the circuit 221 filtering, and image MC motion compensation is applied to the circuit 222 filtering.

Circuit 221 of the filter performs filtering using image MC and MC2 motion compensation, as images MS and MS motion compensation, respectively, in Fig.6, etc. and the intermediate output X, which represents the output of the filter output in the scheme is at 222 filtering.

Circuit 221 of the filter performs filtering using the intermediate output X, and the image MC motion compensation, as images MS and MS motion compensation, respectively, in Fig.6 and so on, and the filter output as the image prediction.

Circuit 211 filter that processes these three keyframes, instead of circuit 72 of the filter may also be provided in the decoding device 1 according to Fig.3 or in the device 101 encoding in Fig.10.

It should be noted that the circuit 221 filtering and circuit 222 of the filter does not have to have the same configuration and may have a different configuration. In addition, the parameters (e.g. α, β) used for filters, can also be made different for each of them, taking into account the characteristics of the input/output obtained before and after filtering.

Circuit 211 filtering can filter not for image motion compensation allocated from the reference frames located in one temporal direction, but for image motion compensation allocated from the three reference frames located in the forward and backward directions.

It should be noted that in the case when the previous and future frames with respect to time of the current frame are used as reference frames, a parameter such as the ratio of output during filtering, can dinamicheski be changed in accordance with the direction of time or distance between the reference frames.

The compressed image information is passed from the device 101 encoding in 1 device decoding through different environments, including recording media, such as optical disk, magnetic disk and storage device type, flash, satellite broadcast, cable TV, Internet & network, mobile telephony.

It should be noted that the above-explained that in the case of encoding and decoding of the extension level image motion compensation extracted from keyframes on the Foundation level. However, image motion compensation can be selected from any level in addition to Foundation level.

For example, it is assumed that the compressed image information forms a three-layer structure having the first level to the third level, where the first level is a Foundation level, which is a layer having the lowest resolution, the second level is a level that has the next lowest resolution, and the third level is the level with the greatest resolution. In this case, when the prediction filter of the third level image motion compensation can be separated from the support frame in the second level, which is not a Foundation level. Of course, the image compensation is designed to provide the Oia can be extracted from the keyframes on the first level, which is a Foundation level.

4. The fourth option exercise

Review decode processing

In addition, image motion compensation can also be extracted from the keyframes on the same level at which to present the image of forecasting, which is to be generated. For example, in the case of a three-tier structure described above, when the prediction filter for the third level of image motion compensation with low resolution can be extracted from the keyframes on the third level.

In Fig.18 shows a diagram illustrating the overview of the decoding process in this case.

In the case of the example shown in Fig.18, the reference frame Ref0 and reference picture Ref1 represent frames on the extension level, which is the same level as the level of the image prediction. In this regard, the resolution for image motion compensation allocated from the respective reference frames will be lower than the resolution of the image prediction.

For example, similarly to the conventional method, the appropriate standard N. 264, position (ranges) corresponding to the target macroblock processing in the reference frames at the extension level, set by the motion vectors of the pixel values in the ranges thinned out with certain cylinder is entom, and allocate image motion compensation. In addition, for example, similarly to the conventional method, the appropriate standard N. 264, the position corresponding to the target macroblock processing in the reference frames at the extension level, set by the motion vectors, and the ranges that are installed in the center of the regulations and which is smaller than the size of the target macroblock processing, allocate, as the image motion compensation. Of course, you can use any other method.

That is, the resolution of image motion compensation, which shall be allocated, may be lower than the resolution of the image prediction, and the method of their selection may be arbitrary. In addition, image motion compensation can be selected from any level.

Image motion compensation with low resolution allocated thus subjected to the filtering processing, including conversion with increasing frequency, similarly to the processing in the other cases described above, and generate the image prediction that has the required permission.

Level, which must be generated by image motion compensation, is simply the other, and the configuration of the encoding device 1 in this case, in principle, similar to the configuration in the case explained with reference to the IG.3-6. In this regard, the storage device 19 of the frame contains frames on the extension level, and the allocation scheme 71 reads the image on the extension level of the storage device 19 of the frame.

The explanation of the flow of the decoding process

In accordance with this, the flow of the decoding process and the decoding process without losses also performed using the method, in principle, similar to the method for the case explained with reference to the flowchart of the sequence of operations in Fig.7 and 8. Flow forecasting process by filtering performed by the circuit 64 forecasting by filtering in this case is explained with reference to the block diagram of the sequence of operations in Fig.19. This block diagram the sequence of operations corresponds to the block diagram of the sequence of operations shown in Fig.9.

When you begin the forecasting process by filtering, at step S341, the circuit 71 allocation allocates the image motion compensation with low resolution of the keyframes on the extension level. Processing steps S342-S350 is performed in a manner analogous to the processing in steps S42-S50 in Fig.9, respectively, and the image prediction generated similarly to the case shown in Fig.9.

When the processing at step S350 is finished, the processing prediction by filtering ends. Then process the TCU returns to the step S28 in Fig.8, and the processing proceeds to step S29.

Thus, even when image motion compensation allocated from staff on the level of expansion, the decoding device 1 may generate the image prediction with high resolution and high accuracy, it is more efficient using temporal correlation, included in the signal sequence in a moving image, and can improve the coding efficiency. In addition, since the image motion compensation can be performed so that they have a lower resolution than the image prediction, for example, the decoding device 1 can reduce the amount of image information, which should be read from the storage device frame, and can improve the coding efficiency by preventing the increased workload associated with processing such as encoding and decoding.

The explanation of the flow of the encoding process

It should be noted that the configuration device 101 encoding corresponding to the decoding device 1 in this case, in principle, similar to the case explained with reference to Fig.10-12. In this regard, the storage device 122 of the frame are frames on the extension level, and processing 123 definition mode reads the image on the extension level of remember what its device 19 of the frame.

Even in this case, as in the case of Fig.11 and 12, the configuration circuit 145 forecasting by filtering process 123 definition mode and circuit 155 forecasting by filtering scheme 125 prediction/motion compensation is also similar to the configuration schema 64 forecasting by filtering shown in Fig.6. In this respect, as in the case of device 1 decoding circuit 71 allocation allocates the image motion compensation of the frame-level extensions.

In accordance with this, the flow of the encoding process or process definition mode, perform in the same way that, in principle, similar to the case explained with reference to the flowchart of the sequence of operations shown in Fig.13 and 14. Flow forecasting process by filtering performed by the circuit 155 forecasting by filtering in this case is explained with reference to the block diagram of the sequence of operations in Fig.20. This block diagram the sequence of operations corresponds to the block diagram of the sequence of operations shown in Fig.15.

When you begin the forecasting process by filtering, at step S451, similarly to step S151 in Fig.15, scheme 151 detection of the motion vector detected motion vectors. At step S452 scheme 71 allocation allocates the image compensation with low resolution is the group of keyframes on the extension level. Processing steps S453-S461 is similar to the processing in steps S153-S161 shown in Fig.15, respectively, and the image prediction generate similar to that shown in Fig.15.

When the processing at step S461 is finished, the forecasting process by filtering ends. Then, the processing returns to step S109 in Fig.13, and the process continues to step S110.

Thus, even when image motion compensation allocated from staff on the level of expansion, the device 101 encoding can generate the image prediction with higher resolution and higher accuracy, more effectively using temporal correlation, included in the signal sequence in a moving image, and can improve the coding efficiency. In addition, since the image motion compensation can be performed so that they have a lower resolution than the image prediction, for example, the device 101 encoding can reduce the amount of image information, which should be read from the storage device frame, and can improve the coding efficiency by preventing an increase in the load on the processing such as encoding and decoding.

Thus, the above method can also be applied in the case to the financing and decoding image information, using a single-level structure, which does not take into account the spatial scalability. That is, the above-described method can also be applied to encoding and decoding in accordance with the standard H. 264/AVC.

It should be noted that, as described above, when allocate image motion compensation, the resolution can be adjusted to a low resolution values. Thus, the selection of image motion compensation can be performed on many levels. In this regard, the processing prediction by filtering requires to determine the difference between the image motion compensation. Thus, by this time you must make the resolution of the respective image motion compensation consistent with each other.

The sequence of processing described above can be performed using hardware or software. In this case, the sequence processing can be performed, for example, a personal computer, as shown in Fig.21.

In Fig.21 CPU (Central processing unit) 501 of the personal computer 500 executes various processes in accordance with the program stored in the ROM (permanent memory) 502, or in accordance with the program loaded in a RAM (random access memory) is module 513 conservation. In the RAM 503 also includes, where appropriate, data and so forth necessary for the CPU 501 to execute various processing.

The CPU 501, ROM 502, and the RAM 503 are connected to each other via a bus 504. The interface 510 I/o is also connected to the bus 504.

Module 511 input, keyboard, mouse and so on, the module 512 output that includes a display, such a display such as CRT (cathode ray tube) or LCD (liquid crystal display), a speaker and so on, the module 513 preserve, consisting of a hard disk, etc. and module 514 data consisting of a modem, etc., connected to the interface 510 input/output. Module 514 data handles the transmission of data across a network, including the Internet.

The actuator 515 is also connected to the interface 510 I/o, in accordance with necessity, and the removable medium 521 such as a magnetic disk, optical disk, magneto-optical disk or semiconductor storage device that is installed in the actuator 515, accordingly. A computer program read from the removable medium 521 record set in module 513 preserve, in accordance with need.

When the processing sequence described above is executed using software, a program constituting the software, installing from the network or from novtel the record.

As shown in Fig.21, the recording media is not just, for example, from a removable medium 521 records, which are supplied separately from the main body of the device for delivery of programs to users and on which is recorded a program, such as a magnetic disk (including a flexible disk), optical disk (including CD-ROM (permanent memory on the CD-ROM) and DVD (digital versatile disk), magneto-optical disk (including MD (mini-disc), or a semiconductor storage device, but also consists of a ROM 502, a hard disk, included in the module 513 conservation, etc. that supply users in the state, pre-installed in the main body of the device, and on which is recorded the program.

It should be noted that the program executed by the computer may be a program in which processes are performed in time sequence in accordance with the procedure described here, or may be a program in which processes are executed in parallel or at necessary points in time, for example, on call.

In addition, as used here, the steps describing the program recorded on the recording media include, of course, the processes performed in time series in the order described here, and also include ina processes, not necessarily processed in a time sequence, but performed in parallel or separately.

In addition, when used herein, the term system refers to a General device, consisting of a set of devices.

In addition, the configuration explained above, as one device (or processing unit) may be divided so as to build a variety of devices (or processing unit). Conversely, the configuration explained above, as a set of devices (or processing unit) may be collected to build a single device (or processing unit). In addition, of course, a different configuration than the one described above, can be added to the configuration of each device (or each processing unit). In addition, the area configuration of a specific device (or processing unit) may be included in the configuration of another device (or another processing unit), if the configuration or operation of the entire system, essentially, is the same. That is, embodiments of the present invention should not be limited to the above-described variants of implementation, and various changes can be made without departing from the essence of the present invention.

For example, the decoding device 1 or device 101 encoding described above can be applied in any e the trip device. Their examples are explained below.

In Fig.22 shows a block diagram illustrating an example configuration of the main part of the television receiver uses the decoding device 1, in which the present invention is applied.

The television receiver 1000, shown in Fig.22, includes terrestrial tuner 1013, the video decoder 1015, circuit 1018 video processing circuit 1019 generate graphs, circuit 1020 control panel and panel 1021 display.

The terrestrial tuner 1013 receives the wave signal broadcast terrestrial analog broadcast via an antenna, demodulates it, receives the video signal and supplies it to the video decoder 1015. The video decoder 1015 performs the decoding process for the video signal supplied from the terrestrial tuner 1013, and delivers the received digital component signal in the circuit 1018 video processing.

Scheme 1018 video processing performs a processing such as noise removal, for the video data supplied from the video decoder 1015, and delivers the received video data in the schema 1019 generation graphics.

Scheme 1019 generation graphics generates video data of the program, which should be displayed on the panel 1021 display, the image data obtained in the course of the process, which is based on the application received through the network, etc. and gives the aerovane video data or image data in the schema 1020 control panel. In addition, the circuit 1019 generate graphs also performs appropriate processing such as the generation of a video (graphics), for display on the screen used by the user to select an item, and so on, puts it on the video program to obtain video data and supplies the obtained video data to the schema 1020 control panel.

Circuit 1020 control panel controls panel 1021 display based on the data transmitted from the circuit 1019 generation graphics, and provides a display of the video program or various screens described above, the panel 1021 display.

Panel 1021 display formed of LCD (liquid crystal display), etc., and displays the video program, etc. in accordance with the control performed by the circuit 1020 control panel.

In addition, the television receiver 1000 also includes a circuit 1014 A/D (analog-digital) Converter audio circuit 1022 processing an audio signal, the circuit 1023 remove the echo/sound-synthesis, circuit 1024 audio amplifier and speaker 1025.

The terrestrial tuner 1013 demodulates the received wave signal is broadcast for reception of video and audio. The terrestrial tuner 1013 delivers the received audio signal in the circuit 1014 A/D Converter output.

Scheme 1014 A/D p is OBRAZOVATEL audio signal processing A/D conversion for audio, supplied from the terrestrial tuner 1013, and delivers the received digital audio signal in the circuit 1022 processing audio.

Circuit 1022 processing audio does some processing, such as removing noise from audio data supplied from the circuit 1014 A/D Converter, and supplies the obtained audio data to the schema 1023 remove the echo/sound-synthesis.

Scheme 1023 remove the echo/sound-synthesis delivers the audio data transmitted from the circuit 1022 processing an audio signal, the circuit 1024 audio amplifier.

Scheme 1024 audio amplifier performs processing D/A conversion and processing gain for the audio data supplied from the circuit 1023 remove the echo signal synthesis, and regulates them up to a certain volume before the sound output from the speaker 1025.

In addition, the television receiver 1000 also includes a digital tuner 1016 and decoder 1017 MPEG.

The digital tuner 1016 receives broadcast wave signal of the digital broadcast (terrestrial digital broadcast, BS (satellite broadcast)/C8 (satellite data) digital broadcast) via an antenna, demodulates it, receives the MPEG-TS (transport stream of the Expert group on the moving image), and supplies it to the decoder 1017 MPEG.

The decoder 1017 MPEG executes descrambling is the MPEG-TS, supplied from the digital tuner 1016, and a thread that includes program data that should be reproduced (must be seen and heard). The decoder 1017 MPEG decodes the audio packets forming the selected stream, and transmits the received audio data in the circuit 1022 processing of the audio signal. In addition, the decoder 1017 MPEG decodes the video packets forming the stream, and delivers the received video data in the schema 1018 video processing. In addition, the decoder 1017 MPEG transmits data to the EPG (electronic program guide), isolated from MPEG-TS to the CPU 1032 through a path that is not shown on the drawing.

The television receiver 1000 uses the decoding device 1 described above, as the decoder 1017 MPEG that decodes videopoker described above. It should be noted that MPEG-TS transmitted from the station broadcast, etc. that has been encoded using the device 101 encoding.

As in the case of device 1 decoding the decoder 1017 MPEG filters images of many of the reference planes at the Foundation level to generate image prediction in the current block on the extension level. In accordance with this decoder 1017 MPEG can more effectively use the components of the signal in the sequence of images than a spatial filter with a transform with increasing frequency. The result of the image prediction may have a higher spatial frequency components, than the image prediction generated by conventional forecasting with a transform with increasing frequency which uses the image of the current frame on the Foundation level, while the remains of the prediction can be reduced. That is, the amount of code for the image to be encoded at the level of expansion can be reduced, and thus it becomes possible to improve coding efficiency.

In addition, this prediction filtering is not accessing the decoded images at the level of expansion in different time frames. Thus, the amount of processing required for coding, capacity retention, the amount of information read from the storage device, etc. can be reduced, and costs needed to implement, can be reduced. In addition, the energy consumption can also be reduced.

Video data supplied from the decoder 1017 MPEG, subjected, as in the case of video data supplied from the video decoder 1015, certain processing circuit 1018 processing of the video signal and the generated video data, etc. impose on them with schema 1019 generation graphics. The resulting data is fed into the panel 1021 display through circuit 1020 control panel and from brehaut their image.

The audio data supplied from the decoder 1017 MPEG, subjected, as in the case of audio data, supplied from the circuit 1014A/D Converter, a specific process by using the circuit 1022 processing an audio signal, is fed into the circuit 1024 audio amplifier through a circuit 1023 remove the echo/sound-synthesis, and subjected to processing D/A conversion or processing gain. Therefore, the audio data, the volume of which was adjusted to a certain value, the output through the speaker 1025.

In addition, the television receiver 1000 also includes a microphone 1026 and circuit 1027 A/D Converter.

Scheme 1027 A/D Converter receives the audio signal of the user who captured using the microphone 1026 provided in the television receiver 1000, for use with audio negotiations, performs the A/D conversion for the received audio signal and supplies the obtained digital audio data to the schema 1023 remove the echo/sound-synthesis.

When sound data of a user (user A) of the television receiver 1000 were submitted from the circuit 1027 A/D Converter circuit 1023 remove the echo/sound-synthesis performs the removal of the echo signal from the audio data of the user a, and provides the output audio data received, for example, by combining with other audio data, through the thunder of govorili 1025, through the scheme 1024 audio amplifier.

In addition, the television receiver 1000 also includes an audio codec 1028, the internal bus 1029, SDRAM (synchronous dynamic random access memory) 1030, a storage device 1031 type flash, CPU 1032, interface 1033 USB (universal serial bus) and a network interface 1034.

Scheme 1027 A/D Converter receives the audio signal of the user who is removed from a microphone 1026, which is provided in a television receiver 1000, for use in an audio conversation, performs the A/D conversion for the received audio signal and transmits the digital audio data to the audio codec 1028.

The audio codec 1028 converts the audio data transmitted from the circuit 1027 A/D Converter data in a specific format for transmission over the network, and delivers them to the network interface 1034 via the internal bus 1029.

Network interface 1034 connected to the network via a cable connected to the network output 1035. Network interface 1034 transmits audio data transferred from the audio codec 102, for example, another device connected to the network. In addition, the network interface 1034 receives, for example, audio data transmitted from another device connected through the network, through the network output 1035, and submits them to the audio codec 1028 via the internal bus 1029.

The audio codec 1028 converts the audio is data, filed through the network interface 1034, data in a specific format, and delivers them into the scheme 1023 remove the echo/sound-synthesis.

Scheme 1023 remove the echo/sound-synthesis performs the removal of the echo signal for the audio data transferred from the audio codec 1028, and provides the output audio data received, for example, by combining with other audio data intended for output through the speaker 1025, via a scheme of 1024 audio amplifier.

In SDRAM 1030 contains various data necessary for processing of the CPU 1032.

In the storage device 1031 type flash contains the program performed by the CPU 1032. The program contained in the storage device 1031 type flash, read CPU 1032 at certain points in time, such as turning on the television receiver 1000. Storage device 1031 type flash also contains EPG data obtained by the digital broadcast data received from a server through a network, and so on

For example, in a storage device 1031 type flash contains MPEG-TS, which includes data content received from a server over the network using the control CPU 1032. Storage device 1031 type flash delivers the MPEG-TS decoder 1017 MPEG via the internal bus 1029, using, for example, the control CPU 1032.

The decoder 1017 handles MPEG MPEG-T, as in the case of MPEG-TS transmitted from the digital tuner 1016. Thus, the television receiver 1000 can receive data content comprising video data, audio data, etc. through the network, decode the data content using a decoder 1017 MPEG, to display the video content and audio output.

In addition, the television receiver 1000 also includes a module 1037 receiving light, which adopts infrared light signal transmitted from the remote control 1051 remote control.

Module 1037 receiving light receives infrared light from the remote control 1051 remote control and outputs a control code indicating the content of operation of the user obtained by demodulation in the CPU 1032.

CPU 1032 executes the program stored in the storage device 1031 type flash, and manages the overall operation of the television receiver 1000, in accordance with the management code that is transmitted from the module 1037 receiving light, etc., the CPU 1032 is connected to each module of the television receiver 1000 through the circuit which is not shown in the drawing.

Interface 1033 USB transmits and receives data in the external device of the television receiver 1000 and from him, which is connected via a USB cable connected to the connector 1036 USB. Network interface 1034 connected to the network via a cable connected to the network connector 1035 and also transmits and principles which comprehends other details, in addition to the audio data from various devices connected through the network.

When using the device 1 decoding in decoder 1017 MPEG television receiver 1000 can obtain a decoded image with high resolution without increasing the load during decoding video packets forming the thread. That is, the television receiver 1000 can improve the coding efficiency by preventing the increase of the load.

In Fig.23 shows a block diagram illustrating an example configuration of the main part of the mobile phone, which uses the decoding device 1 and the device 101 encoding with which the present invention is applied.

The mobile phone 1100, shown in Fig.23, includes a module 1150 main control performed with full control over individual modules, module 1151 circuit of the power source module 1152 operations management input module 1153 encoder image module 1154 camera interface module 1155 LCD control, decoder 1156 image module 1157 MUX/demux module 1162 recording/playback module 1158 schemes modulation/demodulation and the audio codec 1159. They are mutually connected via the bus 1160.

In addition, the mobile phone 1100 includes a button 1119 operations, the camera 1116 CCD (charge coupled device), idcore the metallic display 1118, module 1123 save module 1163 circuit of the transmission/reception antenna 1114, a microphone (mic) 1121 and the speaker 1117.

When the user performing the operation, presses the end call and power button module 1151 circuit power source supplies electric power to each module of the battery pack, launching, thus, the mobile phone 1100, so that he gets the opportunity to work.

The mobile phone 1100 performs various operations in different modes, such as audio call mode and data transfer mode, such as sending and receiving audio, sending and receiving e-mail and image data, the image capture and data entry, on the basis of the control module 1150 main control consisting of CPU, ROM, RAM and so on

For example, audio mobile phone 1100 converts, using the audio codec 1159, the audio signal captured by the microphone (MIC) 1121 in the digital audio data, performs processing spread spectrum for digital audio data using the module 1158 schemes modulation/demodulation, and performs the digital-to-analogue conversion and the process of frequency conversion using the module 1163 scheme of transmission/reception. The mobile phone 1100 transmits the transmission signals obtained by the conversion processing in the base station, which is not shown in the drawing, through ant the nnu 1114. The signal (audio signal) transmitted to the base station enters the mobile phone on the other end of the call through the telephone network of General use.

In addition, for example, audio mobile phone 1100 amplifies, using the module 1163 diagram of a transmission/reception signal, which was adopted via the antenna 1114, additionally performs the process of frequency conversion and the process of analog-to-digital conversion, performs the inverse process of expanding the range using the module 1158 schemes modulation/demodulation, and converts the resulting signal into an analog audio signal using the audio codec 1159. The mobile phone 1100 outputs the analog audio signal obtained by conversion from the speaker 1117.

In addition, for example, in the case where the electronic mail transfer in the data transfer mode, the mobile phone 1100 receives, using the module 1152 management input operations, the text data of the e-mail address entered when performing operations button 1119 operations. The mobile phone 1100 processes the text data using the module 1150 main control, and displays the resulting data as an image on the liquid crystal display 1118 via the module 1155 LCD control.

In addition, the mobile phone 1100, the generating system is t, using the module 1150 basic management, e-mail, based on the text data received by the module 1152 management input operations based on instructions of the user, and so on, the Mobile phone 1100 performs, using the module 1158 schemes modulation/demodulation, the process of expanding the spectrum for e-mail data, and performs, using the module 1163 diagram of a transmission/reception process of digital to analog conversion and the process of frequency conversion. The mobile phone 1100 transmits the transmission signals obtained through conversion processes, in the base station, which are not shown in the drawing, via the antenna 1114. Signal transmission (e-mail) transmitted to the base station, served in a particular destination through the network, mail server, etc.

In addition, for example, in the case when e-mail take in data transfer mode, the mobile phone 1100 receives, using the module 1163 diagram of a transmission/reception signal transmitted from the base station via the antenna 1114, amplifies it, and, in addition, performs the process of frequency conversion and the process of analog-to-digital conversion. The mobile phone 1100 performs the reverse process of the expansion of the spectrum of the received signal using the module 1158 schemes modulation/demodulation, to restore the original e-mail data. Mob the local phone 1100 displays the recovered data of the electronic mail on the liquid crystal display 1118 via the module 1155 LCD control.

It should be noted that the mobile phone 1100 is also configured to record (save) the received e-mail data in the module 1123 save through the module 1162 recording/playback.

Module 1123 save is any module save rewritable. Module 1123 may represent, for example, a semiconductor memory device such as RAM or built-in storage device type, flash, or may be a hard disk or a removable recording medium such as a magnetic disk, a magnetooptical disk, an optical disk, a USB storage device or memory card. Of course, you can use any other media type.

In addition, for example, when image data transfer in the data transfer mode, the mobile phone 1100 generates, using the camera 1116 CCD, the image data through the image. The camera 1116 CCD includes an optical device, such as a lens and aperture, and a CCD is used as the photoelectric conversion element, removes the image of the object, converts the intensity of received light into an electrical signal and generates image data representing images of the object. CCD camera 1116 encodes the image data using the encoder 1153 image through the module 1154 interface the camera, and converts the image data into data of the encoded image.

The mobile phone 1100 uses the device 101 encoding described above, as the encoder 1153 image, which performs the above described processing. As in the case of device 101 encoding encoder 1153 image uses the prediction filter to generate image prediction. In accordance with this can be obtained an image prediction, which includes higher frequency components than the image prediction generated by the unidirectional prediction, bidirectional prediction or forecasting with a transform with increasing frequency, and which has a small difference from the original image. Therefore, only a small amount of code that must be assigned to residues that may be required, and thus it is possible to improve the coding efficiency. Because the resolution of the reference frames is lower than in the case of unidirectional prediction or bidirectional prediction, which appeal to a frame extension level, the load on the processing, such as saving keyframes in the storage device 122 of the frame, and reading the reference frames from the storage device 122 of the frame will be small. In addition, the prediction filter m which may be performed using, at least two keyframes. Thus, the increase coding efficiency becomes technically feasible, without increasing the complexity of the processing.

It should be noted that at this time, the mobile phone 1100 simultaneously performs, using the audio codec 1159, analog-to-digital conversion of the sound collected by the microphone (MIC) 1121 during shooting images using the camera 1116 CCD, and additionally encodes it.

The mobile phone 1100 multiplexes using the module 1157 muxing/demuxing, encoded image data transmitted from the encoder 1153 images, and digital audio data transferred from the audio codec 1159, using a specific schema. The mobile phone 1100 performs, using the module 1158 schemes modulation/demodulation, the process of expansion of the spectrum of the multiplexed data obtained as a result, and performs, using the module 1163 diagram of a transmission/reception process of digital to analog conversion and the process of frequency conversion. The mobile phone 1100 transmits the transmission signals obtained by the conversion process in the base station, which is not shown in the drawing, via the antenna 1114. The transmission signals (image data) transmitted to the base station, served at the other end of the transmission of data across a network, etc.,

It should be noted that h is about in case when the image data is not required to transmit, the mobile phone 1100 can also display the image data generated by using the camera 1116 CCD, the liquid crystal display 1118 via the module 1155 LCD control, without using the encoder 1153 video.

In addition, for example, in the case where the file data of the moving image, in connection with a simplified home page, etc. must be taken in data transfer mode, the mobile phone 1100 receives, using the module 1163 circuit transmission/reception via the antenna 1114, the signal transmitted from the base station, amplifies it, and additionally performs the process of frequency conversion and the process of analog-to-digital conversion. The mobile phone 1100 performs an operation opposite to the extension of the spectrum for the received signal using the module 1158 schemes modulation/demodulation, to restore the original multiplexed data. The mobile phone 1100 demuxes, using the module 1157 muxing/demuxing, multiplexed data, to divide them into encoded image data and audio data.

The mobile phone 1100 decodes the encoded image data using the decoder 1156 image to generate data of a reproduced moving image, and displays danielacristina the moving image in the liquid crystal display 1118 via the module 1155 LCD control. This allows, for example, to display the data of the moving image included in the moving image file associated with a simplified home page, on the liquid crystal display 1118.

In the mobile phone 1100 uses the decoding device 1 described above, as the decoder 1156 image that performs the processes described above. That is, as in the case of device 1 decoding the decoder 1156 image filters for images with multiple reference planes at the Foundation level to generate image prediction of the current block on the extension level. In accordance with this decoder 1156 image can more effectively use the components of the signal in the sequence of images, the filter spatial discretization with increasing frequency. As a result, the image prediction can use spatial high-frequency components than the image prediction generated by conventional forecasting with a transform with increasing frequency which uses the image of the current frame on the Foundation level, while the remains of the prediction can be reduced. That is, the amount of code for the image intended for coding at the level of expansion can be reduced, and what you can improve coding efficiency.

In addition, this prediction filtering does not apply to the decoded images at the level of extension to spatially different time frames. Thus, the amount of processing required for coding, capacity retention, the amount of information read from the storage device, etc. can be reduced, and costs associated with the embodiment can be reduced. In addition, the energy consumption can also be reduced.

At this time, the mobile phone 1100 simultaneously converts digital audio data into an analog audio signal using the audio codec 1159, and provides its output from the speaker 1117. This allows, for example, to reproduce the audio data included in the moving image file associated with the simplified home page.

It should be noted that as in the case of electronic mail, the mobile phone 1100 can also be configured to maintain a record of the received data associated with a simplified home page, etc. to record (save) in the module 1123 save through the module 1162 recording/playback.

In addition, the mobile phone 1100 can also be analyzed using the basic module 1150 management, two-dimensional code obtained by the camera 1116 CCD, by shooting his image, and receive information recorded in dome the nome code.

In addition, the mobile phone 1100 can communicate with an external device via an infrared data link, using the module 1181 infrared data.

The mobile phone 1100 uses the device 101 encoding, as the encoder 1153 image. Thus, for example, coding, which account for the spatial scalability, when the image data generated by the camera 1116 CCD, encode and transmit, through more effective use of temporal correlation is included in the signal sequence in a moving image. Thus, for example, the coding efficiency can be improved while preventing an increase in load during processing, such as encoding and decoding.

In addition, the mobile phone 1100 uses the decoding device 1, as the decoder 1156 image, allowing, thus, to obtain a decoded image with high accuracy, without increasing the load on the processing during decoding, when, for example, receive data (encoded data) of the moving image file linked to a simplified homepage, etc., That is, the mobile phone 1100 can improve the coding efficiency without preventing the increase of the load.

It should be noted that here, we explained that in the mobile phone 1100 COI the box is used Luggage 1116 CCD. However, in the mobile phone 1100 can be used instead of the camera 1116 CCD image sensor (CMOS image sensor), which uses a CMOS (complementary metal oxide semiconductor). Also, in this case, as in the case of using a camera 1116 CCD, the mobile phone 1100 can take an image of an object and generate image data representing images of the object.

In addition, while the foregoing explanation has been presented in the context of the mobile phone 1100, the decoding device 1 and the device 101 encoding can be used, as in the case of the mobile phone 1100, for example, any device having a function for shooting images or data exchange functions similar to the functions 1100 mobile phone, such as a PDA (pocket PC), smart phone, UMPC (ultra mobile personal computer), notebook or laptop personal computer.

In Fig.24 shows a block diagram illustrating an example configuration of the main part of the recording unit on a hard disk that uses the decoding device 1 and the device 101 encoding, in which the present invention is applied.

The 1200 block write to the hard disk (block write to the hard disk) shown in Fig.24 is a device that stores, in its internal jesd the m drive audio and video programs broadcast included in the wave signal broadcast (television signal) transmitted from a satellite, a terrestrial antenna, etc. that were received by the tuner and which provides for user data saved in time in accordance with the user manual.

The device 1200 entries on the hard disk can select, for example, audio data and video data from a waveform broadcast, decode them properly and store them on the internal HDD. In addition, the device 1200 entries on the hard disk can also get audio or video data from another device, for example, through a network, decode them accordingly and save the decoded data on the internal hard disk.

In addition, the device 1200 entries on the hard disk can decode audio data and video data recorded, for example, on the internal hard drive to submit them to the monitor 1260, show them the image on the screen of the monitor 1260, and output the sound through a speaker of the monitor 1260. In addition, the device 1200 entries on the hard disk can also decode, for example, audio data and video data extracted from a waveform broadcast received through the tuner, or AUD is about the data and video data, received from another device via the network, to submit them to the monitor 1260, show them the image on the screen of the monitor 1260, and output the sound through a speaker of the monitor 1260.

Of course, also possible, and other operations.

As shown in Fig.24, the device 1200 entries on the hard disk drive includes a module 1221 of the reception module 1222 demodulator, demultiplexer 1223, audio decoder audio 1224, the video decoder 1225, and the module 1226 device management account. The device 1200 entries on the hard disk, in addition, includes a storage device 1227 EPG data, the storage device 1228 programs, random access memory 1229, Converter 1230 display module 1231 control OSD (menu display on the screen), the module 1232 display control module 1233 recording/playback, D/A Converter 1234 and module 1235 data.

In addition, the Converter 1230 display includes a video encoder 1241. Module 1233 recording/reproduction includes the encoder 1251 and the decoder 1252.

Module 1221 receiving receives an infrared signal from a remote controller (not shown), converts them into an electrical signal and outputs it to the module 1226 device management account. Module 1226 control the recording device, for example, consists of a microprocessor, etc., and performs various processing in accordance with the program is Moi, stored in a storage device 1228 program. At this time, the module 1226 control recorder uses an online storage device 1229 in accordance with the necessity.

Module 1235 data connected to the network, and performs processing of communication with another device via the network. For example, module 1235 data managed by the module 1226 control the recording device to communicate with a tuner (not shown) and to output, in the main, the control signal select channel in the tuner.

Module 1222 demodulator demodulates the signal supplied from the tuner, and outputs it to the demultiplexer 1223. The demultiplexer 1223 further demultiplexes the data fed from the module 1222 demodulation, audio data, video data, and EPG data, and outputs them to the audio decoder audio 1224, the video decoder 1225, and in the module 1226 control the recording device, respectively.

Audio decoder audio 1224 decodes the input audio data and outputs the decoded audio data in the module 1233 recording/playback. The video decoder 1225 decodes the input video data and outputs the decoded video data Converter 1230 display. Module 1226 control the recording device transmits the input EPG data in the memory device 1227 EPG data, to save EPG data.

Converter 1230 display encodes, using the video encoder 1241, video, transferred the data from the video decoder 1225, or module 1226 control the recording device, video data, for example, on a NTSC (national television system Committee standards), and outputs them to the module 1233 recording/playback. In addition, the Converter 1230 display converts the screen size of video data supplied from the video decoder 1225 or module 1226 device management entries in the size corresponding to the size of the monitor 1260, converts the video data into video data according to the NTSC scheme, using the video encoder 1241, converts them to analog signals and outputs them to the module 1232 display control.

Running module 1226 control recorder module 1232 display control superimposes the OSD signal output from the module 1231 control OSD (on-screen display) on the video signal input from the transducer 1230 display, and displays it on the display of the monitor 1260, for his display.

The audio output from the audio decoder 1224, which were converted into an analog signal using a D/A Converter 1234, also served in the monitor 1260. The monitor 1260 outputs audio through its built-in speaker.

Module 1233 recording/reproduction includes a hard disk as the recording media on which recording video data and audio data, etc.

Module 1233 recording/playback encodes, using the encoder 1251, for example, audio data, submitting the successes of the audio decoder 1224. In addition, the module 1233 recording/playback encodes, using the encoder 1251, video data transmitted from the video encoder 1241 Converter 1230 display. Module 1233 recording/playback combines the coded audio data and coded video data using a multiplexer. Module 1233 recording/playback performs channel coding on the received composite data, amplifies them, and writes the data to the hard disk via write head.

Module 1233 recording/playback reproduces the data recorded on the hard disk, through the playback head, amplifies them and separates them into audio data and video data using a demultiplexer. Module 1233 recording/playback decodes audio data and video data using the decoder 1252.

Module 1233 recording/playback performs D/A conversion of the decoded audio data and outputs it to a speaker of the monitor 1260. In addition, the module 1233 recording/playback performs D/A conversion of the decoded video data and displays them on the display of the monitor 1260.

Module 226 controls the recording device reads the latest EPG data from the storage device 1227 EPG data on the basis of instructions of the user transmitted from the infrared signal from the remote control, which is taken through the module 1221, and baking the t EPG data in the module 1231 control OSD. Module 1231 control OSD receives image data that match EPG data, and outputs them to the module 1232 display control. Module 1232 display control displays the video data submitted from the module 1231 control OSD on the display of the monitor 1260, to display them. This allows you to display EPG (electronic program TV broadcasts) on the display of the monitor 1260.

In addition, the device 1200 entries on the hard disk can also receive various data such as video data, audio data, and EPG data supplied from another device via a network such as the Internet.

Module 1235 data transfer control using the control module 1226 recording device for receiving encoded data such as video data, audio data, and EPG data transmitted from another device through a network, and delivers them to the module 1226 device management account. Module 1226 control burner delivers, for example, encoded data of the received video data and audio data in the module 1233 recording/playback to save them on your hard drive. At this time, the module 1226 control capture device and module 1233 recording/playback, if necessary, can perform processing such as re-encoding.

In addition, the module 1226 control the recording device decodes the coded data received is data and audio data and supplies the obtained video data Converter 1230 display. Converter 1230 display processes the video data transferred from the module 1226 control the recording device, similarly to the processing of video data supplied from the video decoder 1225, and supplies the resulting video data to the monitor 1260 through the module 1232 display control to display their images.

In addition, together with the image display module 1226 device management account may submit the decoded audio data to the monitor 1260 through the D/A Converter 1234 and output the sound through the loudspeaker.

In addition, the module 1226 control the recording device decodes the coded data received EPG data and supplies the decoded EPG data in the memory device 1227 EPG data.

The device 1200 entries on the hard disk, such as described above, uses the decoding device 1, as each of the video decoder 1225, the decoder 1252, and a built-in decoder module 1226 device management account. That is, as in the case of device 1 decoding, the video decoder 1225, the decoder 1252, and a built-in decoder module 1226 control the recording device to perform filtering of images consisting of multiple reference planes at the Foundation level, to generate image prediction of the current block is at the level of extensions.

Accordingly, the video decoder 1225, the decoder 1252, and the triplex decoder module 1226 device control write enable more efficient use of signal components in the image sequence, than a spatial filter of increasing the sampling rate. As a result, the image prediction can have a spatial high-frequency components than the image prediction generated by conventional forecasting with a transform with increasing frequency which uses the image of the current frame on the Foundation level, while the remains of the prediction can be reduced. That is, the amount of code for the image to be encoded at the level of expansion can be reduced, and thus it is possible to improve coding efficiency.

In addition, when such a prediction filter does not perform a reference to the decoded image on the extension level at different time frames. Thus, the amount of processing required for coding, capacity retention, the amount of information read from the storage device, etc. can be reduced, and costs associated with the embodiment can be reduced. In addition, can also be reduced energy consumption.

In accordance with this, the device 1200 entries on the hard disk can obtain a decoded image with high accuracy, without increasing the load on the processing during decoding, when, for example, a tuner or module 1235 transfer d is the R accepts video data (encoded data) or when the module 1233 recording/playback reproduces video data (encoded data) from the hard disk. That is, the device 1200 entries on the hard disk can improve the coding efficiency, without preventing the increase of the load.

In addition, the device 1200 record to the hard drive uses your device 101 encoding as the encoder 1251. Accordingly, as in the case of device 101 encoding, the encoder 1251 receives the image prediction, which includes higher frequency components than the image prediction generated by bidirectional prediction or forecasting with a transform with increasing frequency, and which has a small difference from the original image. Therefore, only a small amount of code that must be appointed for the remainder, can be reduced, and thus it is possible to increase the coding efficiency. Because the resolution of the reference frames is lower than in the case of unidirectional prediction or bidirectional prediction in which the appeal to the personnel at the extension level, the load on the processing, such as saving keyframes in the storage device 122 of the frame and the reading of the reference frames from the storage device 122 of the frame is small. In addition, the prediction filtering may be performed using at least two reference frames. Thus, this increase in efficiency is Yunosti coding becomes technically feasible without increasing the complexity of the processing.

In accordance with this, the device 1200 entries on the hard disk performs encoding, which account for the spatial scalability, when, for example, the coded data recorded on the hard disk, through more effective use of temporal correlation is included in the sequence of signals in a moving image, and can therefore, for example, to improve the coding efficiency, without preventing the load increases during processing, such as encoding and decoding.

It should be noted that, while the device 1200 entries on the hard disk, which records video and audio data on the hard disk, described above, of course, any type of recording media may be used. For example, even the recording device, which uses a different recording media, except for the hard disk, such as a mass storage device type, flash, optical disk or the recording can also use the decoding device 1 and the device 101 encoding, as in the case of device 1200 entries on the hard disk drive described above.

In Fig.25 shows a block diagram illustrating an example configuration of the main part of the camera that uses the decoding device 1 and the device 101 encoding, in which the present invention is applied.

The camera 1300, shown in Fig.25, Sneem is t the image of the object and displays the object image on the LCD 1316 or writes it to the media 1333 records as image data.

Block 1311 lens provides light incidence (that is, the video object on the CCD/CMOS 1312. CCD/CMOS 1312 is an image sensor that uses a CCD or CMOS sensor, converts the intensity of received light into an electric signal and supplies it to the module 1313 signal processing chamber.

Module 1313 signal processing camera converts the electric signal supplied from the CCD/CMOS 1312, color difference signals Y, Cr and Cb and submits them to the module 1314 processing the image signal. Under the control of the controller 1321 module 1314 processing the image signal performs specific image processing to the image signal supplied from module 1313 signal processing chamber, or encodes the image signal using the encoder 1341. Module 1314 signal processing of the image takes the encoded data generated by encoding the image signal, the decoder 1315. In addition, the module 1314 processing the image signal receives display data generated by the module 1320 display the on screen menu (OSD), and supplies them to the decoder 1315.

In the above-described processing module 1313 signal processing camera uses DRAM (dynamic random access memory) 1318 connected via the bus 1317, and stores the image data, encoded data obtained by to the financing of the image data, etc. in the DRAM 1318, if necessary.

The decoder 1315 decodes the coded data transmitted from a module 1314 processing the image signal, and supplies the obtained image data (decoded image data) in the LCD 1316. In addition, the decoder 1315 delivers the display data supplied from module 1314 signal processing of the image in the LCD 1316. LCD 1316 combines the image of the decoded image data supplied from the decoder 1315, and the image data of the display, respectively, and displays the resulting composite image.

Under the control of the controller 1321 module 1320 display menu on-screen displays the data display, such as a menu screen generated from characters, letters or numbers, and the icon in the module 1314 processing the image signal via the bus 1317.

The controller 1321 performs various processing based on the signal indicating the content of the commands generated by the user using module 1322 operations, and also controls the module 1314 processing the image signal, the DRAM 1318, the external interface 1319, 1320 display menu on the screen, drive 1323 media, etc. via the bus 1317. ROM 1324 type flash contains programs, data, etc. necessary for the controller 1321, to perform various processing.

For example, the controller 1321 may encode the image data stored in the DRA 1318, or to decode the coded data stored in the DRAM 1318, from the name of a module 1314 processing the image signal or the decoder 1315. At this time, the controller 1321 may perform the encoding process or decoding, using a scheme similar to the scheme for encoding or decoding module 1314 processing the image signal or the decoder 1315, or may perform the encoding process or decoding, using a scheme that is not supported by the module 1314 signal image processing or the decoder 1315.

In addition, for example, when the instruction to start printing the image was generated by the module 1322 operations, the controller 1321 reads the image data from the DRAM 1318 and submits them to the printer 1334 connected to the external interface 1319, via the bus 1317, for printing.

In addition, for example, in the case where the instruction to record the image was produced from module 1322 operations, the controller 1321 reads the encoded data from the DRAM 1318 and delivers them to the carrier 1333 record set in the drive 1323 the recording media via the bus 1317, to save them.

Media 1333 record represents, for example, any removable media write read and overwritten, such as a magnetic disk, a magnetooptical disk, optical disk or semiconductor storage device. Media 1333 records can also PR is dostavljati a, of course, any type of removable media and may be a tape device, disk or memory card. Of course, you can also use contactless IC card, etc.

In addition, the drive 1323 the recording media and the media 1333 records may also be integrally formed, for example, removable storage media, such as a built-in hard drive of a magnetic disk or SSD (solid state drive).

The external interface 1319, for example, consists of connector I/o USB, etc. and it is connected to the printer 1334 when you want to print the image. In addition, the drive 1331 is connected to the external interface 1319, in accordance with necessity, and removable media 1332, such as magnetic disk, optical disk or a magnetooptical disk set, respectively, so that a computer program read from it, is installed in ROM 1324 type flash, in accordance with need.

In addition, the external interface 1319 includes a network interface which is connected to a certain network, such as LAN or the Internet. The controller 1321 may read the encoded data from the DRAM 1318 in accordance with the instruction, for example, from a module 1322 operations, and delivering them to another device connected through the network from the external interface 1319. In addition, the controller 1321 may receive, via the external interface 139 coded data or image data, supplied from another device via the network, and keep them in the DRAM 1318 or to submit them to the module 1314 signal processing of the image.

The camera 1300, such as described above, uses the decoding device 1, as the decoder 1315. That is, as in the case of device 1 decoding, the decoder 1315 performs filtering of images from multiple reference planes at the Foundation level, to generate image prediction of the current block on the extension level. In accordance with this, the decoder 1315 can more effectively use the components of the signal in the sequence of images than a spatial filter, which increases the sampling rate. As a result, the image prediction can have a spatial high-frequency components than the image prediction generated by the conventional predictive transform with increasing frequency which uses the image of the current frame on the Foundation level, while the remains of the prediction can be reduced. That is, the amount of code for the image to be encoded at the level of expansion can be reduced, and thus it is possible to improve coding efficiency.

In accordance with this, the camera 1300 can improve the coding efficiency, without preventing the helicene load, for example, for image data generated by the CCD/CMOS1312, when encoded data of video data read from the DRAM 1318 or the media 1333 recording or when the encoded data of video data received via the network.

In addition, the camera 1300 is used, the device 101 encoding as the encoder 1341. As in the case of device 101 encoding the encoder 1341 can get the image prediction, which includes a greater number of high-frequency components than the image prediction generated using bi-directional prediction or forecasting with a transform with increasing frequency, and which has a small difference from the original image. Therefore, only a small amount of code that must be assigned to residues can be reduced, and thus it is possible to improve the coding efficiency. Because the resolution of the reference frames is lower than in the case of unidirectional prediction or bidirectional prediction in which the appeal to the personnel at the extension level, the load of processing, such as storing reference frames in a storage device of the frame 122 and the read reference frames from the storage device 122 to be minor. In addition, the prediction filtering can be performed using minicamera, the two keyframes. Thus, this increase coding efficiency may be technically feasible, without increasing the complexity of the processing.

In accordance with this camera 1300 performs encoding, which account for the spatial scalability, when, for example, the coded data recorded in the DRAM 1318 or the media 1333 recording, or when the coded data is provide in another device, through more effective use of temporal correlation is included in the sequence of signals in a moving image, and can therefore, for example, to improve the coding efficiency, without preventing increase the load on processes such as encoding and decoding.

It should be noted that the method of decoding the decoding device 1 can be applied to the decoding process performed by the controller 1321. Similarly, the encoding device 101 encoding may be applied to the encoding process performed by the controller 1321.

In addition, the image data captured by a camera 1300 can represent data of a moving image or a still image.

Of course, the decoding device 1 and the device 101 encoding can also be used in another device or system than the device described above.

In addition, rosmarinifolia are arbitrary. The present invention can be applied, for example, the macroblock having any dimensions, as shown in Fig.26. For example, the present invention can be applied not only to the normal macroblock size of 16×16 pixels, but also to the extended macroblock (macroblock extensions), such as a macroblock of size 32×32 pixels.

In Fig.26, in the upper part, the macroblocks consisting of 32×32 pixels, which are divided into blocks (sections) of 32×32 pixels, 32×16 pixels, 16×32 pixels, and 16×16 pixels, presented in sequence, starting from the left. In addition, in the middle part, the blocks of 16×16 pixels, which are divided into blocks of size 16×16 pixels, 16×8 pixels, 8×16 pixels and 8×8 pixels shown in sequence, starting from the left. In addition, at the bottom, blocks consisting of 8×8 pixels, which are divided into blocks of size 8×8 pixels, 8×4 pixels, 4×8 pixels and 4×4 pixels, are presented in sequence, starting from the left.

Thus, the macroblocks of size 32×32 pixels can be processed in blocks of 32×32 pixels, 32×16 pixels, 16×32 pixels, and 16×16 pixels shown in the upper part.

Blocks of size 16×16 pixels shown on the right side in the upper part, can be processed similarly to scheme H. 264/AVC, in blocks of 16×16 pixels, 16×8 pixels, 8×16 pixels and 8×8 pixels shown in the medium is part.

The block size of 8×8 pixels shown on the right side in the middle part, may be processed similarly to scheme H. 264/AVC, blocks of 8×8 pixels, 8×4 pixels, 4×8 pixels and 4×4 pixels shown in the lower part.

The above-described blocks can be classified into the following three hierarchical levels. Thus, the blocks of 32×32 pixels, 32×16 pixels, and 16×32 pixels, shown in the upper part of Fig.26, are called the blocks of the first hierarchical level. The block of 16×16 pixels shown on the right side in the upper part, and the blocks of 16×16 pixels, 16×8 pixels and 8×16 pixels, shown in the middle part, called blocks at the second hierarchical level. The block of 8×8 pixels shown on the right side in the middle part, and the blocks of 8x8 pixels, 8×4 pixels, 4×8 pixels and 4×4 pixels shown in the lower part, called blocks at the third hierarchical level.

If we take the above-described structure of the hierarchical levels, in respect of units, equal to or less than the block of 16×16 pixels, larger blocks can be defined as their swarnapuri, while maintaining compatibility with the scheme of H. 264/AVC.

For example, the decoding device 1 and the device 101 encoding can be performed with the opportunity to generate an image prediction for each hierarchical level. In addition, for example, the device 1 decterov the Oia and the device 101 encoding can be performed with use of the image prediction, which generate at the first hierarchical level, which has a larger block size than the second hierarchical level, the second hierarchical level.

Macroblocks for which must be completed encoding using a relatively large block size, such as the first hierarchical level and a second hierarchical level, do not include the relatively high frequency components. In contrast, the macroblocks for which the encoding should be performed using the unit of relatively small size, such as, at the third hierarchical level are considered, include relatively high-frequency components.

Thus, the image prediction generate separately according to the individual hierarchical levels with different block sizes, making, therefore, technically feasible improvements in performance encoding that is suitable for the local characteristics of the image.

The list of numbers of the reference positions

1 - device for decoding 12 - decoding scheme is lossless, 15 - diagram of the adder 19 is a storage device of the frame 21 is a diagram of the prediction/motion compensation, 41 - schema definition forecasting, 51 - scheme of forecasting, 64 - prediction scheme by filtration, 71 - selection scheme, 72 diagram fil the radio 81 diagram of the difference calculation, 82 - scheme with a transform with increasing frequency, 83 diagram of the filter of low frequencies, 84 diagram of the gain control, 85 - filter circuit of high frequency, 86 diagram of the gain control, 87 diagram of the adder 88 scheme with a transform with increasing frequency, 89 diagram of the adder 101 is a device for encoding, 112 - buffer layout change - 123-circuit mode, 125 diagram prediction/motion compensation, 126 diagram of the intra-frame prediction, 145 - prediction scheme by filtering, 155 - prediction scheme by filtration, 211 - circuit filter.

1. The imaging device containing:
the allocator that is intended to perform motion compensation using, as reference frames, frames formed of decoded images, and using the motion vectors in the images that have been encoded, and for selecting image motion compensation having a lower resolution than the image prediction from the reference frames corresponding to the image prediction; and
a means of generating image prediction that is designed to perform the filtering process on at least two images of the motion compensation allocated by the allocator, using the correlation in the direction of the lie is neither, which is included in the first image motion compensation and the second image motion compensation, generating, thus, the image prediction with a higher resolution than the image motion compensation.

2. The imaging device under item 1,
in which a means of generating image prediction made with the possibility of adding a second filtered image motion compensation to the first image motion compensation.

3. The imaging device under item 1,
in which a means of generating image prediction made with the ability to add images, representing the high-frequency component, to the first image motion compensation.

4. The imaging device under item 1,
in which the images that were coded were hierarchically decomposed into images on multiple levels, with different resolutions, and image have been encoded,
in which, when the decoding level with high resolution, the allocator uses, as reference frames, frames at a level having a lower resolution than on the level, and selects the image motion compensation from a reference frame at a level having a lower resolution, and
which means what about the generation of the image prediction generates image prediction at the level of high resolution, performing a filtering process for the image motion compensation allocated from the keyframes on the level with low resolution.

5. The imaging device under item 1,
in which a means of generating image prediction includes
the conversion tool resolution, designed for converting the resolution difference image between multiple image motion compensation allocated by the allocator, and increase resolution,
the filter medium low frequency designed to filter low frequency to a Delta image, whose resolution has been increased conversion tool permissions
the means of the high-pass filter designed to filter high frequency to the image obtained by applying the filter of low frequencies in the medium of the first filter, and
means summation intended for summation image obtained by applying filter low frequency filter medium low frequency, and the image obtained by applying a high-pass filter in the filter medium high frequency for one of the many image motion compensation allocated by the allocator, and to generate image prediction.

6. The processing device from the images on p. 5,
which means summation sums the image obtained by applying filter low frequency filter medium low frequency, and the image obtained by applying a high-pass filter means filter high frequency image motion compensation, selected from the previous frame relative to the time-image prediction.

7. The imaging device under item 1, additionally containing:
the tool reception designed for reception flag identification identifying whether to generate an image prediction through unidirectional prediction, whether to generate an image prediction through bidirectional prediction, or image prediction should be generated through the filtering process performed by means of generating image prediction; and
the determination tool, designed to determine whether to generate an image prediction through unidirectional prediction, whether to generate an image prediction through bidirectional prediction, or image prediction should be generated through the filtering process with reference to the flag identification, adopted by the means of reception.

8. The processing device from the images on p. 7, additionally contains:
the tool unidirectional prediction, designed to perform unidirectional prediction using a variety of image motion compensation, and to generate image prediction;
the tool bidirectional prediction, designed to perform bi-directional prediction using a variety of image motion compensation, and to generate image prediction.

9. The imaging device under item 1, additionally containing:
the a decoder designed to decode the encoded image; and
the generating tool that is intended to be a summation of the image decoded by the decoding means, and the image prediction and to generate a decoded image.

10. The imaging device under item 1, additionally containing:
the detection means designed to detect the motion vectors on the basis of images and the original image that is an image used to encode each of the images get by performing local decoding on the basis of a residual signal indicating a difference between the original image and the image prediction.

11. Eliminate the STV image processing on p. 10,
in which, when the decoding level with high resolution, the allocator uses, as reference frames, frames at a level having a lower resolution than on the level, and selects the image motion compensation from a reference frame at the level of having a low resolution, using the motion vectors detected by the detection means at the level of having a low resolution, and
in which a means of generating generates the image prediction at the level of the high resolution by performing a filtering process for the image motion compensation allocated from the keyframes on the level with low resolution.

12. The imaging device according to p. 10,
in which a means of generating includes
the conversion tool resolution, designed for converting the resolution difference image between multiple image motion compensation allocated by the allocator, and increase resolution,
the filter medium low frequency, intended for the application of low frequency to the difference image whose resolution has been increased conversion tool permissions
the means of the high-pass filter designed to filter high frequency to the image obtained by applying the filter, n is scoi frequency filter media low frequency, and
means summation intended for summation image obtained by applying the filter low frequency filter medium low frequency, and the image obtained by applying a high-pass filter means filter high frequency, for one of the many image motion compensation allocated by the allocator, and generating image prediction.

13. The imaging device according to p. 12,
which means summation sums the image obtained by applying filter low frequency filter medium low frequency, and the image obtained by applying a high-pass filter means filter high frequency image motion compensation, selected from the previous frame relative to the time-image prediction.

14. The imaging device according to p. 10, further comprising:
a means of encoding used to encode the original image that is an image designed for encoding, and generating an encoded image.

15. The imaging device according to p. 10,
which means encoding ensures the header flag identification that identifies whether the image prediction, cotorobai added to the image, decoded by the decoding device, to be generated through unidirectional prediction, generated through bidirectional prediction, or generated through the filtering process.

16. The method of image processing, comprising:
performing motion compensation using, as reference frames, frames formed of decoded images, and using the motion vectors in the images that have been encoded, and produce at least two image motion compensation having a lower resolution than the image prediction from the reference frames corresponding to the image prediction; and
perform filtering processing for compensating at least two selected image motion compensation using the correlation in the time direction that is included in the first image motion compensation and the second image motion compensation, generating, thus, the image prediction, which has a higher resolution than the at least two image motion compensation.

17. The method according to p. 16, in which the filtered second image motion compensation is added to the first image motion compensation.

18. The method according to p. 16, in which the image representing the high-frequency com is onent, add to the first image motion compensation.

19. The method according to p. 16, optionally, in which
detects motion vectors based on the images and the original image, and each of the images get by performing local decoding on the basis of a residual signal indicating a difference between the original image and the image prediction.



 

Same patents:

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering. The invention can be implemented in a conditional access content broadcast system where it is desirable to identify and take measures against receiving equipment, applied when sharing control words. Owing to the requirement that receiving equipment used in the system transmits to a transmission station a conditional access content message at a precisely defined time, the invention provides a method through which a server identifies receiving equipment participating in the sharing of control words and prevents said receiver from further accessing said content.

EFFECT: effective protection of transmitted content.

12 cl, 2 dwg

FIELD: physics.

SUBSTANCE: method of forming a digital watermark-certified electronic colour image includes, at the transmitting side, scaling digital image data to a standard size, applying discrete Fourier transform (DFT) to said data and determining the DFT amplitude and simultaneously recording the phase, selecting local areas of the amplitude spectrum such that the positions of said centres coincide with the positions of nonzero key K values, selecting maximum values of the amplitude spectrum in each local area and correcting said values under the condition of maintaining acceptable image quality, placing the corrected maximum values of the amplitude spectrum at the centre of each local area by replacing initial values of the amplitude spectrum at said positions, embedding an identification key in the image, performing an operation to restore amplitude symmetry and, using the phase stored earlier, performing inverse DFT to obtain digital data with an embedded watermark.

EFFECT: improved protection of an electronic image certified by a digital watermark of the creator of the image from deliberate violation and improved image quality.

4 cl, 4 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering. A method of coding coefficients associated with a block of video data during a video coding process, the method comprising determining statistics that indicate a probability of each of the X and Y coordinates comprising a given value, when a scanning order associated with the block comprises a first scanning order, wherein X and Y coordinates indicate a horizontal position and a vertical position, respectively, of the last non-zero coefficient within the block according to the scanning order, wherein the scanning order comprises one of a first scanning order and a second scanning order, which is different from the first scanning order; and coding the X and Y coordinates based on the statistics when the scanning order comprises a first scanning order, and interchanging the X and Y coordinates and coding the interchanged X and Y coordinates based on the statistics when the scanning order comprises a second scanning order.

EFFECT: high efficiency of coding the position of a last significant coefficient.

62 cl, 17 dwg

FIELD: information technology.

SUBSTANCE: method of compression of graphic file by fractal method using ring classification of segments, in which the graphic file is split into rank regions and domains, and for each rank region the domain and the corresponding affine transformation is found, that best approximates it to the appropriate rank region, and using the obtained values of the domain parameters, comprising their coordinates, the coefficients of the affine transformations, the values of brightness and contrast, the archive is formed, and classification of domains and rank regions are introduced, based on the allocation in them of the "rings" and the calculation of the mathematical expectation of pixel intensity of these "rings", which enables to reduce the complexity of the phase of correlation of the segments and to accelerate compression.

EFFECT: reduced time of compression of the graphic file by fractal method.

3 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to presentation of content recommendations. A computer-implemented system (100) for presenting content to a subscriber comprises: a detection component (104) for detecting an inactive subscriber in a session of a website portal; a tracking component (114) for tracking interaction of the inactive subscriber with the session content; and an analysis component (202) for analysing subscriber browsing behaviour, extrapolating subscriber experience level based on the analysis, and suggesting content as session content, wherein the subscriber experience level is extrapolated by tracking how often a subscriber selects Help and other menu categories; and a content component (116) for obtaining and presenting new content targeted to the inactive subscriber during the session based on the tracked interaction of the inactive subscriber with the session content, wherein the session content is new content as part of the session content based on the subscriber experience level.

EFFECT: providing dynamic tracking of information on a new or inactive user on a website and fast presentation of targeted content back to the user to maintain the interest of the user in the website.

13 cl, 9 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to an apparatus and a method of creating an electronic programme guide from programme information stored in an electronic programme guide database for a plurality of channels. The apparatus for creating an electronic programme guide from programme information stored in an electronic programme guide database for a plurality of channels comprises a control unit, a display unit, a time selection unit comprising a user input interface, a programme information search unit and a determining device. The control unit further includes a displayed information updating unit. An electronic programme guide displays elements relating to programmes accessible on a plurality of channels at a given moment in time. When the given time changes, the electronic programme guide updates elements corresponding to channels on which a programme which is accessible at a new time is different from a programme accessible at the previous time.

EFFECT: reduced amount of data required to update an electronic programme guide.

17 cl, 7 dwg

FIELD: physics, video.

SUBSTANCE: invention relates to means of storing and transporting encoded video data. The method includes assembling encoded video data into a video file fragment comprising sub-track fragments, which comprise a plurality of hierarchically related encoded video images of the encoded video data, arranged continuously in the decoding order within a corresponding sub-track fragment, wherein the hierarchically related encoded video images each correspond to a common hierarchical layer; outputting data; receiving a request in accordance with a streaming protocol which specifies the sub-track fragment. In the method, the request comprises a hypertext transfer protocol (HTTP) partial GET request which specifies a byte range corresponding to the sub-track fragment; outputting the hierarchically encoded video images of the sub-track fragment.

EFFECT: retrieving encoded images of a specific hierarchical layer in a video fragment using a single request.

44 cl, 9 dwg, 3 tbl

FIELD: physics.

SUBSTANCE: method comprises phases, during which video bit stream containing a sequence of image frames is received, the error occurrence is determined in the display frame segment, time distribution and spatial distribution of the named error is identified by means of the motion vector information and internal prediction relating a segment, affected by the error, the quality of video bit stream is assessed on the basis of the named error distribution.

EFFECT: improvement of accuracy of assessment at the expense of identification of error spatial distribution in the display frame segment.

18 cl, 16 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to computer engineering. A method of encoding video which comprises determining a transformation unit, i.e., a data unit in which a current coding unit from among coding units is transformed, the coding units being data units in which a current video image is encoded, wherein the size of the transformation unit is determined independent of the size of at least one prediction unit which is a data unit in the current coding unit; transforming data of the current coding unit based on the transformation unit to encode the data of the current coding unit; and outputting the encoded data of the current coding unit, encoding mode information which indicates an encoding mode of the encoded data of the current coding unit, and transformation index information which indicates a structure of the transformation unit, as a bit stream.

EFFECT: high rate of encoding and decoding data.

15 cl, 23 dwg, 1 tbl

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to media content information processing. The method comprises: providing (S1) a pseudo-identifier of a media codec used to encode said media content; providing (S2) additional processing information which defines a post-decoding instruction for processing the decoded media content obtained by decoding said encoded media content; and joint organisation (S3) of said pseudo-identifier and said additional processing information in a file in order to stop decoding said encoded media content by a conventional media terminal which does not recognise said pseudo-identifier, and enable decoding of said encoded media content using said media codec and post-decoding processing of said decoded media content using said additional processing information by a media terminal which recognises said pseudo-identifier.

EFFECT: high efficiency of processing data.

20 cl, 13 dwg

FIELD: information technology.

SUBSTANCE: method of compression of graphic file by fractal method using ring classification of segments, in which the graphic file is split into rank regions and domains, and for each rank region the domain and the corresponding affine transformation is found, that best approximates it to the appropriate rank region, and using the obtained values of the domain parameters, comprising their coordinates, the coefficients of the affine transformations, the values of brightness and contrast, the archive is formed, and classification of domains and rank regions are introduced, based on the allocation in them of the "rings" and the calculation of the mathematical expectation of pixel intensity of these "rings", which enables to reduce the complexity of the phase of correlation of the segments and to accelerate compression.

EFFECT: reduced time of compression of the graphic file by fractal method.

3 dwg

FIELD: physics.

SUBSTANCE: method comprises making each array element in an image sensor from one "R, G, B radiation colour brightness to code" converter, which performs parallel synchronous conversion of radiation of three colours analogue video signals R, G, B into three codes. The frame image digitisation apparatus includes an objective lens, an image sensor comprising an array of elements, three switch units, three register units and a control signal generator, wherein each switch unit includes the same number of encoders as converters.

EFFECT: reduced cross dimensions of array elements in an image sensor, which enables to reduce the frame format size or increase resolution of the image sensor.

6 dwg, 1 tbl

FIELD: physics.

SUBSTANCE: disclosed is a frame image digitisation apparatus. The disclosed apparatus comprises a lens in the focal plane of which there is an image sensor having an array of elements, a control signal generator and three register units, the outputs of which are the outputs of the digitisation apparatus. Each array element consists of a converter for converting radiation of colours R, G, B into three codes. Images are input into the sensor, the number of said images being equal to the number of array elements and the number of colours R, G, B of analogue-to-digital converters (ADC).

EFFECT: high image frame resolution owing to conversion of three colours R, G, B into codes using one converter.

4 dwg, 2 tbl

FIELD: physics.

SUBSTANCE: apparatus comprises a lens, an image detector having an array situated in the focal plane of the lens, the array having elements which are converters for converting radiation to codes based on the frame resolution number 106, each having an opaque housing in the front part of which, in a partition wall, there is a microlens, on the optical axis of which and at an angle of 45° thereto semitransparent micromirrors are arranged in series and rigidly mounted based on the number of bits per code, each preceding micromirror transmitting to the next micromirror radiation flux with half the strength.

EFFECT: high speed of frame digitisation.

1 tbl, 4 dwg

FIELD: physics.

SUBSTANCE: apparatus comprises a lens, an image detector which includes an array of elements based on the frame resolution number 106, situated in the focal plane of the lens and having three groups of outputs of colour codes R, G, B, includes three register units and a control signal generator which outputs from the first output pulses with frame frequency (25 Hz), connected to the control inputs in array elements, and from the second output pulses with code sampling frequency, connected in parallel to the second control inputs of the first through third register units.

EFFECT: high frame resolution by making array element converters of brightness of radiation of colours R, G, B - three codes, which synchronously output codes of three colours R, G, B.

5 dwg, 1 tbl

FIELD: physics.

SUBSTANCE: disclosed is a method of obtaining a structural image of a biological object in optical coherence tomography. The method includes breaking down a source colour vide frame into non-overlapping spatial blocks consisting of more than one pixel. A structural image is obtained via small-angle raster scanning in the arm of an optical coherence tomography sample. The obtained image with a size of Piskh bytes is broken down into non-overlapping spatial blocks only on columns; adjacent column blocks are averaged pixel by pixel to form a new image with a size of Pstl bytes; the new image is broken down into non-overlapping spatial blocks only on rows; adjacent row blocks are averaged pixel by pixel to form a resultant image with a size of Pres bytes, and the averaging process is controlled based on an exponential relationship Pstl from the number of averaging column blocks Ustl and Pres from the number of averaging row blocks - Ustr.

EFFECT: high quality of the structural image of a biological object in optical coherence tomography.

7 dwg

FIELD: physics, video.

SUBSTANCE: invention relates to video encoding and decoding, which provides conversion between a spatial region and a conversion region. The method of encoding video includes breaking down a video image into encoding units having a maximum size. The images are encoded based encoding units according to depths obtained by hierarchically breaking down each maximum depth encoding unit, and based on the type of partition defined according to depths of the depth encoding units. The type of partition includes a data unit having the same size as the current encoding unit, and a partial data unit obtained by breaking down the height or width of the current encoding unit. Encoding units are determined according to encoding depths relative to each of the depth encoding units, and encoding units having a tree structure are therefore determined. The encoded data are output.

EFFECT: high efficiency of image compression and, as a result, high efficiency of encoding and decoding video.

15 cl, 15 dwg, 1 tbl

FIELD: physics, video.

SUBSTANCE: invention relates to encoding three-dimensional video signals, and specifically to a transport format used to transport three-dimensional content. The technical result is achieved using a device which is characterised by that it includes a means of generating a stream which is structured into multiple levels: level 0, having two independent layers: a base layer containing video data of a right-side image, and a level 0 extension layer containing video data of a left-side image, or vice versa; level 1, having two independent extension layers: a level 1 first extension layer containing a depth map relating to the image of the base layer, a level 1 second extension layer containing a depth map relating to the image of the level 0 extension layer; level 2, having a level 2 extension layer containing overlapping data relating to the image of the base layer.

EFFECT: high quality of three-dimensional images with a large number of presentations used.

6 cl, 2 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to means of encoding and decoding images. In the method, the motion vector of a reference section has the same shape as the current section and belongs to a reference image which is different from the current image and is broken down in advance as a result of encoding with subsequent decoding into a plurality of sections. When a reference section overlaps a set of reference sections from said plurality of sections of the reference image, said motion vector of the current image section is determined based on a reference motion vector function belonging to a set of reference motion vectors associated with k overlapped reference sections.

EFFECT: high accuracy of predicting the motion vector of an image section.

15 cl, 6 dwg

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to predictive motion vector predictive encoding/decoding of moving pictures. The moving picture encoding apparatus includes a primary candidate reference motion vector determination unit which sets N primary candidate reference motion vectors, a degree of reliability calculation unit which calculates the reliability of each primary candidate reference motion vector which quantitatively represents effectiveness in motion vector prediction of the block to be decoded, using encoded or decoded picture information, a reference motion vector determination unit selects M (M<N) secondary candidate reference motion vectors in accordance with the degree of reliability of N primary candidate reference motion vectors, a motion vector encoding unit calculates a predictive motion vector of the block to be encoded using M secondary candidate reference motion vectors with high reliability.

EFFECT: improved efficiency of predicting and encoding moving pictures.

16 cl, 14 dwg

FIELD: physics.

SUBSTANCE: method comprises phases, during which video bit stream containing a sequence of image frames is received, the error occurrence is determined in the display frame segment, time distribution and spatial distribution of the named error is identified by means of the motion vector information and internal prediction relating a segment, affected by the error, the quality of video bit stream is assessed on the basis of the named error distribution.

EFFECT: improvement of accuracy of assessment at the expense of identification of error spatial distribution in the display frame segment.

18 cl, 16 dwg

Up!