{"title": "On the accuracy of l1-filtering of signals with block-sparse structure", "book": "Advances in Neural Information Processing Systems", "page_first": 1260, "page_last": 1268, "abstract": "We discuss new methods for the recovery of signals with block-sparse structure, based on l1-minimization. Our emphasis is on the efficiently computable error bounds for the recovery routines. We optimize these bounds with respect to the method parameters to construct the estimators with improved statistical properties. We justify the proposed approach with an oracle inequality which links the properties of the recovery algorithms and the best estimation performance.", "full_text": "On the accuracy of (cid:96)1-\ufb01ltering of signals with\n\nblock-sparse structure\n\nAnatoli Juditsky\u2217\n\nFatma K\u0131l\u0131nc\u00b8 Karzan\u2020\n\nArkadi Nemirovski\u2021\n\nBoris Polyak\u00a7\n\nAbstract\n\nWe discuss new methods for the recovery of signals with block-sparse structure,\nbased on (cid:96)1-minimization. Our emphasis is on the ef\ufb01ciently computable error\nbounds for the recovery routines. We optimize these bounds with respect to the\nmethod parameters to construct the estimators with improved statistical proper-\nties. We justify the proposed approach with an oracle inequality which links the\nproperties of the recovery algorithms and the best estimation performance.\n\nIntroduction\n\n1\nSuppose an observation y \u2208 Rm is available where\n\ny = Ax + u + D\u03be.\n\n(1)\nHere A is a given m \u00d7 n sensing matrix, x \u2208 Rn is an unknown vector, u is an unknown (determin-\nistic) nuisance parameter, known to belong to a certain set U \u2282 Rm, D \u2208 Rm\u00d7m is known noise\nintensity matrix, and \u03be \u2208 Rm is random noise with standard normal distribution.\nWe aim to recover a linear transformation w = Bx of the signal x, where B is a given N \u00d7 n matrix,\nunder the assumption that w is block-sparse. Namely, the space W(= RN ) where w \u201clives\u201d is\nrepresented as W = Rn1 \u00d7 ...\u00d7RnK , so that w = Bx \u2208 RN is a block vector: w = [w[1]; ...; w[K]]\nwith blocks w[k] = B[k]x \u2208 Rnk, 1 \u2264 k \u2264 K, where B[k], 1 \u2264 k \u2264 K are nk \u00d7 n matrices.\nThe s-block-sparsity of w means that at most a given number s of the blocks u[k], 1 \u2264 k \u2264 K, are\nnonzero.\nTo motivate the interest for the presented model, let us consider two examples.\n\nTracking of a singularly perturbed linear system Consider a discrete-time linear system\n\nz[i] = Gz[i \u2212 1] + w[i] + F \u03b7[i],\n\ni = 1, 2, ..., z[0] \u2208 Rd,\n\nwhere F \u03b7[i] are random perturbations with \u03b7[i] being i.i.d. standard normal vectors \u03b7[i] \u223c N (0, Id),\nand G, F \u2208 Rd\u00d7d are known matrices. We assume that the perturbation vectors w[i] \u2208 Rd i =\n1, 2, ... are mostly zeros, but a small proportion of w[i] are nonvanishing unknown vectors in Rd.\nSuppose that we are given the linear noisy observation y \u2208 Rm, such that y = A[z[0]; ...z[K]] + \u03c3\u03be,\nwhere the matrix A \u2208 Rm\u00d7d(K+1) and the noise intensity \u03c3 > 0 are known, and \u03be \u223c N (0, Im).\nk=1, w[k] \u2208 Rd, and\nGiven y, our objective is to recover the sequence of perturbations w = [w[k]]K\nthe trajectory z of the system.\n\n\u2217LJK, Universit\u00b4e J. Fourier, B.P. 53, 38041 Grenoble Cedex 9, France, Anatoli.Juditsky@imag.fr\n\u2020Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA, fkilinc@andrew.cmu.edu\n\u2021Georgia Institute of Technology, Atlanta, Georgia 30332, USA, nemirovs@isye.gatech.edu\nResearch of the second and the third authors was supported by the Of\ufb01ce of Naval Research grant #240667V.\n\u00a7Institute of Control Sciences of Russian Academy of Sciences, Moscow 117997, Russia,\n\nboris@ipu.rssi.ru\n\n1\n\n\fTo \ufb01t the tracking problem into the basic framework let us decompose z = x + \u03b6, where x =\n[x[0]; ...; x[K]] with x[i] = Gx[i \u2212 1] + w[i], x[0] = z[1], and \u03b6 = [\u03b6[0]; ...; \u03b6[K]] with \u03b6[i] =\nG\u03b6[i \u2212 1] + F \u03b7[i], \u03b6[0] = 0. Then\n\ny = Ax + [A\u03b6 + \u03c3\u03be],\n\nwhere the distribution of A\u03b6 + \u03c3\u03be is normal with zero mean and the covariance matrix D2 =\nAV AT + \u03c32I. Here the Kd \u00d7 Kd covariance matrix V of \u03b6 has the block structure with blocks\n\n(cid:96)\u2227k(cid:88)\n\ni=1\n\nV k,(cid:96) = Cov(\u03b6[k], \u03b6[(cid:96)]) =\n\nG(cid:96)\u2212iF F T (GT )k\u2212i,\n\nwith(cid:80)0\n\n1 = 0, by convention.\n\nImage reconstruction with regularization by Total Variation (TV)\n[21, 7] Here one looks to\nrecover an image Z \u2208 Rn1\u00d7n2 from a blurred noisy observation y: y = Az + \u03c3\u03be, y \u2208 Rm, where\nz = Col(Z) \u2208 Rn, n = n1n2, A \u2208 Rm\u00d7n is the matrix of discrete convolution, \u03c3 > 0 is known,\nand \u03be \u223c N (0, Im). We assume that the image z may be decomposed as z = x + v, where v is a\n\u201cregular component\u201d, which is modeled by restricting v to belong to the set V of \u201csmooth images\u201d;\nlet w = Bx \u2208 Rn\u00d72, be the (discretized) gradient of the x-component at the points of the grid.\nIn this example Bx naturally splits into 2-dimensional blocks and TV is nothing but the sum of (cid:96)2\nnorms of these blocks. We suppose that w is (nearly) sparse. When denoting u = Av we come to\nthe observation model y = Ax + u + D\u03be, with u \u2208 U = AV, D = \u03c3I, and \u03be \u223c N (0, Im).\n\nThe recovery routines we consider are based on the block-(cid:96)1 minimization, i.e., the estimate (cid:98)w(y)\nof w = Bx is (cid:98)w = B(cid:98)x(y), where(cid:98)x(y) is obtained by minimizing the norm(cid:80)K\nk=1 (cid:107)B[k]z(cid:107)(k) over\nsignals z \u2208 Rn with Az \u201c\ufb01tting,\u201d in certain precise sense, the observations y. Above, (cid:107) \u00b7 (cid:107)(k) are\ngiven in advance norms on the spaces Rnk where the blocks of w take their values.\nIn the sequel we refer to the given in advance collection B = (B, n1, ..., nK,(cid:107) \u00b7 (cid:107)(1), ...,(cid:107) \u00b7 (cid:107)(K))\nas to the representation structure. Given such a structure B and matrix A, our goal is to understand\nhow well one can recover the s-block-sparse transform Bx by appropriately implemented block-(cid:96)1\nminimization.\n\nRelated Compressed Sensing research Our situation and goal form a straightforward extension\nof the usual block sparsity Compressed Sensing framework. Indeed, the standard representation\nstructure B = In, nk = 1, (cid:107)\u00b7(cid:107)(k) = |\u00b7|, 1 \u2264 k \u2264 K = n, leads to the standard Compressed Sensing\nsetting \u2013 recovering a sparse signal x \u2208 Rn from its noisy observations (1) via (cid:96)1 minimization. With\nthe same B = In and nontrivial block structure {nk,(cid:107) \u00b7 (cid:107)(k)}K\nk=1, we arrive at block-sparsity and\nrelated block-(cid:96)1 minimization routines considered in numerous recent papers. There is a number\nof applications where block-sparsity seem to arise naturally (see, e.g., [10] and references therein).\nSeveral methods of estimation and selection extending the plain (cid:96)1-minimization to block sparsity\nwere proposed and investigated recently. Most of the related research is focused so far on block\nregularization schemes \u2013 Lasso-type algorithms\n\n(cid:8)(cid:107)Az \u2212 y(cid:107)2\n\n2 + \u03bbL[1,2](z)(cid:9) , L[1,2](z) =\n\nK(cid:88)\n\nk=1\n\n(cid:107)z[k](cid:107)2,\n\n(cid:98)x(y) \u2208\n\nArgmin\n\nz=[z[1];...;z[K]]\u2208Rn=Rn1\u00d7...\u00d7RnK\n\n(cid:107) \u00b7 (cid:107)2 being the \u201cusual\u201d (cid:96)2-norm on Rnk. In particular, the huge literature on plain Lasso has a\nsigni\ufb01cant counterpart on group Lasso, see, e.g., [1, 2, 8, 9, 10, 11, 16, 19, 20, 22, 23], and references\ntherein. Another classical Compressed Sensing estimator, Dantzig Selector, is studied in the block-\nsparse case in [12, 17]. The available theoretical results allows to bound the errors of recovery in\nterms of magnitude of the observation noise and \u201cs-concentration\u201d of the true signal x (that is, its\nL[1,2] distance from the space of signals with at most s nonzero blocks). Typically, these results deal\nwith the quadratic risks of estimation and rely on a natural block analogy (\u201cBlock RIP,\u201d see, e.g.,\n[10]) of the celebrated Restricted Isometry property for the sensing matrix A, introduced by Cand\u00b4es\nand Tao [5], or on a block analogy [18] of the Restricted Eigenvalue property from [3].\n\nContributions of this work To the best of our knowledge, the conditions used when studying\ntheoretical properties of block-sparse recovery (with a notable exception of the Mutual Block In-\ncoherence condition of [9]) are unveri\ufb01able. The latter means that given the matrix A, one cannot\n\n2\n\n\fanswer in any reasonable time if the (block-) RI or RE property holds with given parameters. While\nthe ef\ufb01cient veri\ufb01ability of a condition is by no means necessary for a condition to be meaningful\nand useful, we believe also that veri\ufb01ability has its value and is worthy of being investigated. In par-\nticular, the veri\ufb01ability property allows us to design new recovery routines with explicit con\ufb01dence\nbounds for the recovery error and optimize these bounds with respect to the method parameters.\nThus, the major novelty in what follows is the emphasis on veri\ufb01able conditions on A and the repre-\nsentation structure which guarantee good recovery of Bx from noisy observations of Ax, provided\nthat Bx is nearly s-block-sparse, and the observation noise is low. In this respect, this work extends\nthe results of [15, 13, 14], where (cid:96)1-recovery of the \u201cusual\u201d sparse vectors was considered (in the\n\ufb01rst two papers \u2013 in the case of uncertain-but-bounded observation errors, and in the third \u2013 in the\ncase of Gaussian observation noise). We propose new routines of block-sparse recovery which ex-\nplicitly utilize the veri\ufb01ability certi\ufb01cate \u2013 the contrast matrix, and show how these routines may be\ntuned to attain the best performance bounds.\nThe rest of the manuscript is organized as follows.\nIn Section 2 we give the detailed problem\nstatement and introduce the family Qs,q, 1 \u2264 q \u2264 \u221e, of conditions which underly the subsequent\ndevelopments. Then in Section 2.3 we introduce the recovery routines and provide the bounds for\ntheir risks. We discuss the properties of conditions Qs,q in Section 3. In particular, in Section 3.1 we\nshow how one can ef\ufb01ciently verify (the strongest from the family Qs,q) condition Qs,\u221e. Then in\nSection 3.2 we provide an oracle inequality which shows that the condition Qs,\u221e is also necessary\nfor recovery of block-sparse signals in (cid:96)\u221e-norm.\n\n2 Accuracy bounds for (cid:96)1-recovery routines\n\ni=1 |w[k]i|r)\n\n1\n\n1 \u2264 r \u2264 \u221e: (cid:107)w[k](cid:107)r = ((cid:80)nk\n\n2.1 Problem statement and notations\nLet w = Bx \u2208 W = Rn1 \u00d7 ... \u00d7 RnK . To streamline the presentation, we restrict ourselves to\nthe case where all the norms (cid:107) \u00b7 (cid:107)(k) on the factors of the representations are the usual (cid:96)r-norms,\nr , i.e., the representation structures we consider are\nB = (B, n1, ..., nK, (cid:107) \u00b7 (cid:107)r). Let r\u2217 = r\nr\u22121, so that (cid:107) \u00b7 (cid:107)r\u2217 is the norm conjugate to (cid:107) \u00b7 (cid:107)r. A vector\nw = [w[1]; ...; w[K]] from W is called s-block-sparse, if the number of nonzero blocks w[k] \u2208 Rnk\nin w is at most s.\nFor w \u2208 W, we call the number (cid:107)w[k](cid:107)r the magnitude of k-th block in w, and denote by ws the\nrepresentation vector obtained from w by zeroing out all but the s largest in magnitude blocks in\nw (with the ties resolved arbitrarily). For w \u2208 W and 1 \u2264 p \u2264 \u221e, we denote by L[p,r](w) the\n(cid:107)\u00b7(cid:107)p-norm of the vector [(cid:107)w[1](cid:107)r; ...;(cid:107)w[K](cid:107)r], so that L[p,r](\u00b7) is a norm on W with the conjugate\np\u22121. Given a positive integer s \u2264 K, we\nnorm L\u2217\nset Ls,[p,r](w) = L[p,r](ws); note that Ls,[p,r](\u00b7) is a norm on W. When the representation structure\nB of x (and thus the norm (cid:107) \u00b7 (cid:107)r) is \ufb01xed, we use the notation Lp, L\u2217\np, and Ls,p, instead of L[p,r],\nL\u2217\n[p,r], and Ls,[p,r].\nThe recovery problem we are interested in is as follows: suppose we are given an indirect observation\n(cf (1))\n\n[p,r](w) = (cid:107)[(cid:107)w[1](cid:107)(r\u2217); ...;(cid:107)w[K](cid:107)(r\u2217)](cid:107)p\u2217, p\u2217 = p\n\ny = Ax + u + D\u03be\n\nof unknown signal x \u2208 Rn. Here A \u2208 Rm\u00d7n, u + D\u03be is the observation error; in this error, u is\nan unknown nuisance known to belong to a given compact convex set U \u2282 Rm symmetric w.r.t. the\norigin, D \u2208 Rm\u00d7m is known, and \u03be \u223c N (0, Im).\nWe want to recover x and the representation w = Bx of x, knowing in advance that this representa-\ntion is nearly s-block-sparse, for some given s. Speci\ufb01cally, we consider the set\n\nX(s, \u03c5) = {x \u2208 Rn : L1(Bx \u2212 [Bx]s) \u2264 \u03c5}.\n\nA recovery routine is a Borel function(cid:98)x(y) : Rm \u2192 Rn and we characterize the performance of\nsuch a routine by its Lp-risk of recovery (cid:98)w(y) = B(cid:98)x(y) of w = Bx:\n\n= inf {\u03b4 : Prob\u03be {Lp ((cid:98)w(y) \u2212 w) \u2264 \u03b4 \u2200(u \u2208 U, x \u2208 X(s, \u03c5))} \u2265 1 \u2212 \u0001} .\n\nRiskp((cid:98)w(\u00b7)|s, D, \u03c5, \u0001)\n\n3\n\n\fhere 0 \u2264 \u0001 \u2264 1 and 1 \u2264 p \u2264 \u221e. In other words, Riskp((cid:98)w(\u00b7)|s, D, \u03c5, \u0001) \u2264 \u03b4 if and only if there\nnorm of B[(cid:98)x(y) \u2212 x] is \u2264 \u03b4 whenever \u03be \u2208 \u039e, u \u2208 U, and whenever x \u2208 Rn is such that Bx\n\nexists a set \u039e \u2208 Rm of \u201cgood realizations of \u03be\u201d such that Prob{\u03be \u2208 \u039e} \u2265 1 \u2212 \u0001 and the L[p,r]-\n\ncan be approximated by s-block-sparse representation vector within accuracy \u03c5 (measured in the\nL[1,r]-norm).\n\n2.2 Condition Qs,q(\u03ba)\nLet a sensing matrix A and a representation structure B = (B, n1, ..., nK,(cid:107) \u00b7 (cid:107)r) be given, and let\ns \u2264 K be a positive integer, q \u2208 [1,\u221e] and \u03ba > 0. We say that a pair (H,(cid:107)\u00b7(cid:107)), where H \u2208 Rm\u00d7M\nand (cid:107) \u00b7 (cid:107) is a norm on RM , satis\ufb01es the condition Qs,q(\u03ba) associated with the matrix A and B, if\n\n\u2200x \u2208 Rn : Ls,q(Bx) \u2264 s\n\n1\n\nq (cid:107)H T Ax(cid:107) + \u03bas\n\n1\n\nq \u22121L1(Bx).\n\n(2)\n\nThe following is an evident observation.\nObservation 2.1 Given A and a representation structure B, let (H,(cid:107) \u00b7 (cid:107)) satisfy Qs,q(\u03ba). Then\n(H,(cid:107) \u00b7 (cid:107)) satis\ufb01es Qs,q(cid:48)(\u03ba(cid:48)) for all q(cid:48) \u2208 (1, q) and \u03ba(cid:48) \u2265 \u03ba. Besides this, if s(cid:48) \u2264 s is a positive\ninteger, ((s/s(cid:48))\nWhenever (B, n1, ..., nK,(cid:107) \u00b7 (cid:107)r) is the standard representation structure, meaning that B is the\nidentity matrix, n1 = 1, and (cid:107) \u00b7 (cid:107)r = | \u00b7 |, the condition Qs,q(\u03ba) reduces to the condition Hs,q(\u03ba)\nintroduced in [14].\n\nq H,(cid:107) \u00b7 (cid:107)) satis\ufb01es Qs(cid:48),q((s(cid:48)/s)1\u2212 1\n\nq \u03ba).\n\n1\n\n2.3\n\n(cid:96)1-Recovery Routines\n\nWe consider two block-sparse recovery routines.\n\nRegular (cid:96)1 recovery\n\nis given by\n\n(cid:98)xreg(y) \u2208 Argmin\n\nz\n\n(cid:8)L1(Bz) : (cid:107)H T (Az \u2212 y)(cid:107) \u2264 \u03c1(cid:9) ,\n\nwhere H \u2208 Rm\u00d7M , (cid:107) \u00b7 (cid:107) and \u03c1 > 0 are parameters of the construction.\nTheorem 2.1 Let s be a positive integer, q \u2208 [1,\u221e] and \u03ba \u2208 (0, 1/2). Let also \u0001 \u2208 (0, 1). Assume\nthat the parameters H, (cid:107) \u00b7 (cid:107), \u03c1 of the regular (cid:96)1-recovery are such that\n\nA. (H,(cid:107)\u00b7(cid:107)) satis\ufb01es the condition Qs,q(\u03ba) associated with the matrix A and the representation\n\nstructure B;\n\nB. There exists a set \u039e \u2282 Rm such that Prob(\u03be \u2208 \u039e) \u2265 1 \u2212 \u0001 and\n\n(cid:107)H T (u + D\u03be)(cid:107) \u2264 \u03c1 \u2200(u \u2208 U, \u03be \u2208 \u039e).\n\nThen for all 1 \u2264 p \u2264 q and \u03c5 > 0,\n\nRiskp(B(cid:98)xreg(y)|s, D, \u03c5, \u0001) \u2264 (4s)\n\n1\np\n\nPenalized (cid:96)1 recovery is(cid:98)xpen(y) \u2208 Argmin\n\nz\n\n2\u03c1 + s\u22121\u03c5\n1 \u2212 2\u03ba\n\n, 1 \u2264 p \u2264 q.\n\n(cid:8)L1(Bz) + 2s(cid:107)H T (Az \u2212 y)(cid:107)(cid:9) ,\n\nwhere H \u2208 Rm\u00d7M , (cid:107) \u00b7 (cid:107) and a positive integer s are parameters of the construction. The accuracy\nof the penalized recovery is given by the following analogue of Theorem 2.1:\nTheorem 2.2 Let s be a positive integer, q \u2208 [1,\u221e] and \u03ba \u2208 (0, 1/2). Let also \u0001 \u2208 (0, 1). Assume\nthat the parameters H, (cid:107) \u00b7 (cid:107), s of the penalized recovery and a \u03c1 \u2265 0 satisfy conditions A, B from\nTheorem 2.1. Then for all 1 \u2264 p \u2264 q and \u03c5 > 0 we have\n\nRiskp(B(cid:98)xpen(y)|s, D, \u03c5, \u0001) \u2264 2(2s)\n\n1\np\n\n2\u03c1 + s\u22121\u03c5\n1 \u2212 2\u03ba\n\n, 1 \u2264 p \u2264 q,\n\n(5)\n\n(3)\n\n(4)\n\ncf. (4).\n\n4\n\n\f3 Evaluating Condition Qs,\u221e(\u03ba)\n\nThe condition Qs,q(\u03ba), of Section 2.2, is closely related to known conditions, introduced to study\nthe properties of recovery routines in the context of block-sparsity. Let us consider the representation\nstructure with B = In. If the norm (cid:107) \u00b7 (cid:107) in (2) is chosen to be the (cid:96)\u221e-norm, we have the following\nobvious observation:\n\n(!) Let H satisfy Qs,q(\u03ba) and let(cid:98)\u03bb be the maximum of the Euclidean norms of\n\ncolumns in H. Then\n\n\u2200x \u2208 Rn : Ls,q(x) \u2264(cid:98)\u03bbs\n\n1\n\nq (cid:107)Ax(cid:107)2 + \u03bas\n\n1\n\nq \u22121L1(x).\n\nNote that conditions of this kind with \u03ba < 1/2 and (cid:107)\u00b7(cid:107)r = (cid:107)\u00b7(cid:107)2 play a crucial role in the performance\nanalysis of group-Lasso and Dantzig Selector. For example, the error bounds for Lasso recovery,\nobtained in [18] rely upon the Restricted Eigenvalue assumption RE(s, \u03ba) which is as follows: there\nis \u03ba > 0 such that\n\nHence, Ls,1(x) \u2264 \u221a\n\nL2(xs) \u2264 1\nsLs,2(x) \u2264 \u221a\n\n\u03ba(cid:107)Ax(cid:107)2 whenever 3L1(xs) \u2265 L1(x \u2212 xs).\ns\u03ba (cid:107)Ax(cid:107)2 whenever 4Ls,1(x) \u2265 L1(x), so that\n\n\u2200x \u2208 Rn : Ls,1(x) \u2264 s1/2\n\n\u03ba (cid:107)Ax(cid:107)2 +\n\n1\n4\n\nL1(x)\n\n(6)\n\n(observe that (6) is nothing but the \u201cblock version\u201d of the Compatibility condition from [4]).\nThe bad news is that, in general, condition Qs,q(\u03ba), as well as RE and Compatibility conditions,\ncannot be veri\ufb01ed. Speci\ufb01cally, given a sensing matrix A and a representation structure B, it seems\nto be dif\ufb01cult even to verify that a pair (H,(cid:107)\u00b7(cid:107)) satis\ufb01es condition Qs,q(\u03ba) associated with A, B, let\nalone to synthesize such H which satis\ufb01es this condition and results in the best possible error bounds\n(4), (5) for the regular and the penalized (cid:96)1-recoveries. The good news is that when (cid:107) \u00b7 (cid:107)r is the\nuniform norm (cid:107)\u00b7(cid:107)\u221e and, in addition, q = \u221e the condition Qs,q(\u03ba) becomes \u201cfully computationally\ntractable\u201d.1 We intend to demonstrate also that this condition Qs,\u221e(\u03ba) is in fact necessary for the\nbounds of the form (4), (5) to be valid when p = \u221e.\n3.1 Condition Qs,\u221e(\u03ba), case r = \u221e: tractability\nConsider the case of the representation structure B\u221e = (B, n1, ...nK,(cid:107)\u00b7(cid:107)\u221e). We have the following\nresult.\nProposition 3.1 Let (cid:107) \u00b7 (cid:107)(k) = (cid:107) \u00b7 (cid:107)\u221e for all k \u2264 K, and let a positive integer s and reals \u03ba > 0,\n\u0001 \u2208 (0, 1) be given.\n(i) Assume that a triple (H,(cid:107) \u00b7 (cid:107), \u03c1), where H \u2208 RM\u00d7m, (cid:107) \u00b7 (cid:107) is a norm on RM , and \u03c1 \u2265 0, is such\nthat\n\n(!) (H,(cid:107)\u00b7(cid:107)) satis\ufb01es Qs,\u221e(\u03ba) and the set \u039e = {\u03be : (cid:107)H T [u+D\u03be](cid:107) \u2264 \u03c1 \u2200u \u2208 U}\nis such that Prob(\u03be \u2208 \u039e) \u2265 1 \u2212 \u0001.\n\nThen there exist N = n1+...+nK vectors h1, ..., hN in Rm and N\u00d7N block matrix V = [V k(cid:96)]K\n(the blocks V k(cid:96) of V are nk \u00d7 n(cid:96) matrices) such that\n\nk,(cid:96)=1\n\n(cid:18)\n\n(cid:107)V k(cid:96)(cid:107)\u221e,\u221e \u2264 s\u22121\u03ba \u2200k, (cid:96) \u2264 K\n\n(a) B = V B + [h1, ..., hN ]T A,\n(b)\n(here (cid:107)V k(cid:96)(cid:107)\u221e,\u221e = max1\u2264j\u2264n(cid:96) (cid:107)Rowj(V k(cid:96))(cid:107)1, Rowj(M ) being the j-th row of M),\n(c) Prob\n\n(cid:19)\nu\u2208U uT hi + |(D\u03be)T hi| \u2264 \u03c1, 1 \u2264 i \u2264 N}\n(cid:98)H = [h1, ..., hN ], the norm (cid:107) \u00b7 (cid:107)\u221e on RN and \u03c1 form a triple satisfying (!).\n\n(ii) Whenever vectors h1, ..., hN \u2208 Rm and a matrix V = [V k(cid:96)]K\n\nk,(cid:96)=1 satisfy (7), the m \u00d7 N matrix\n\n\u039e+ := {\u03be : max\n\n\u2265 1 \u2212 \u0001.\n\n(7)\n\n1Recall that by Observation 2.1, q = \u221e corresponds to the strongest among the conditions Qs,q(\u03ba) asso-\nciated with A and a given representation structure B and ensures the validity of the bounds (4) and (5) in the\nlargest possible range, 1 \u2264 p \u2264 \u221e, of values of p.\n\n5\n\n\fDiscussion. Let a sensing matrix A \u2208 Rm\u00d7n and a representation structure B\u221e be given, along\nwith a positive integer s, an uncertainty set U, and quantities D and \u0001. Recall that Theorems 2.1, 2.2\nsay that if a triple (H,(cid:107) \u00b7 (cid:107), \u03c1) is such that (H,(cid:107) \u00b7 (cid:107)) satis\ufb01es Qs,\u221e(\u03ba) with \u03ba < 1/2 and H, \u03c1 are\nsuch that for the set\n\n\u039e = {\u03be : (cid:107)H T [u + D\u03be](cid:107) \u2264 \u03c1 \u2200u \u2208 U}\n\nit holds P (\u039e) \u2265 1 \u2212 \u0001, then for all \u03c5 \u2265 0, for the regular (cid:96)1 recovery associated with (H,(cid:107) \u00b7 (cid:107), \u03c1)\nand for the penalized (cid:96)1 recovery associated with (H,(cid:107) \u00b7 (cid:107), s) one has:\n\nRiskp(B(cid:98)x|s, D, \u03c5, \u0001) \u2264 2(2s)\n\n1\np\n\n2\u03c1 + s\u22121\u03c5\n1 \u2212 2\u03ba\n\n, 1 \u2264 p \u2264 \u221e.\n\n(8)\n\nProposition 3.1 states that when applying this result, we lose nothing by restricting ourselves with\ntriples H = [h1, ..., hN ] \u2208 Rm\u00d7N , N = n1 + ... + nK, (cid:107) \u00b7 (cid:107) = (cid:107) \u00b7 (cid:107)\u221e on RN , \u03c1 \u2265 0 which can be\naugmented by an appropriately chosen N \u00d7 N matrix V to satisfy relations (7). In the rest of this\ndiscussion, it is assumed that we are speaking about triples (H,(cid:107) \u00b7 (cid:107), \u03c1) satisfying the just de\ufb01ned\nrestrictions.\nNow, as far as bounds (8) are concerned, they are completely determined by two parameters \u2014 \u03ba\n(which should be < 1/2) and \u03c1; the smaller are these parameters, the better are the bounds. In what\nfollows we address the issue of ef\ufb01cient synthesis of matrices H with \u201cas good as possible\u201d values\nof \u03ba and \u03c1.\nObserve, \ufb01rst, that H = [h1, ..., hN ] and \u03ba should admit an extension by a matrix V to a solution of\nthe system of constraints (7). Let \u00b5U (h) = max\n\nProb(cid:0)\u039e+ =(cid:8)\u03be : \u00b5U (hi) + |(D\u03be)T hi| \u2264 \u03c1, 1 \u2264 i \u2264 N(cid:9)(cid:1) \u2265 1 \u2212 \u0001,\n\nu\u2208U uT h. Note that the restriction\n\n(cid:16) \u0001\n(cid:17)(cid:107)DT hi(cid:107)2\nIgnoring the \u201cgap\u201d between er\ufb01nv(\u0001/2) and er\ufb01nv(cid:0) \u0001\n\nwhere er\ufb01nv(\u00b7) is the inverse error function2, and it is implied by\n\n(cid:17)(cid:107)DT hi(cid:107)2\n(cid:105)\n(cid:1), we can safely model the restriction (9) by\n\n\u03c1 \u2265(cid:104)\n\n\u03c1 \u2265 max\n1\u2264i\u2264N\n\n, 1 \u2264 i \u2264 N.\n\n\u00b5U (hi) + er\ufb01nv\n\n\u00b5U (hi) + er\ufb01nv\n\n(cid:16) \u0001\n\nimplies that\n\nthe system of convex constraints (10). Thus, the set Gs of admissible \u03ba, \u03c1 can be safely approxi-\nmated by the computationally tractable convex set\n\n(cid:105)\n\n,\n\n2N\n\n2N\n\n(cid:104)\n\n(9)\n\n(10)\n\n2\n\n(cid:26)\n\nG\u2217\ns =\n\n(\u03ba, \u03c1) : \u2203H = [h1, ..., hN ] \u2208 Rm\u00d7N , V = [V k(cid:96) \u2208 Rnk\u00d7n(cid:96)]K\n, 1 \u2264 k, (cid:96) \u2264 K\n\nB = BV + H T A, (cid:107)V k(cid:96)(cid:107)\u221e,\u221e \u2264 \u03ba\ns\n\n(cid:17)(cid:107)DT hi(cid:107)2 \u2264 \u03c1, 1 \u2264 i \u2264 N\n\n(cid:16) \u0001\n\nu\u2208U uT hi + er\ufb01nv\nmax\n\n2N\n\n(cid:27)\n\nk,(cid:96)=1 s.t.\n\n3.2 Condition Qs,\u221e(\u03ba), case r = \u221e: necessity\nLet the representation structure B\u221e = (B, n1, ..., nK,(cid:107) \u00b7 (cid:107)\u221e) be \ufb01xed. From the above discussion\nwe know that if, for some \u03ba < 1/2 and \u03c1 > 0, there exist H = [h1, ..., hN ] \u2208 Rm\u00d7N and\nV = [V k(cid:96) \u2208 Rnk\u00d7n(cid:96)]K\nk,(cid:96)=1 satisfying (7), then regular (cid:96)1-recovery with appropriate choice of\nparameters ensures that\n\nRisk\u221e(B(cid:98)xreg|s, D, \u03c5, \u0001) \u2264 2\u03c1 + s\u22121\u03c5\n\n1 \u2212 2\u03ba\n\n.\n\n(11)\n\nWe are about to demonstrate that this implication can be \u201cnearly inverted:\u201d\n\n2i.e., u = er\ufb01nv(\u03b4) means that\n\n1\u221a\n2\u03c0\n\n(cid:82) \u221e\nu e\u2212t2/2dt = \u03b4.\n\n6\n\n\fProposition 3.2 Let a sensing matrix A, an uncertainty set U, and reals \u03ba > 0, \u0001 \u2208 (0, 1) be\ngiven. Assume also that the observation error \u201cis present,\u201d speci\ufb01cally, that for every r > 0, the set\n{u + De : u \u2208 U,(cid:107)e(cid:107)2 \u2264 r} contains a neighborhood of the origin.\n\nGiven a positive integer S, assume that there exists a recovering routine(cid:98)x satisfying an error bound\n\n\u2200(x \u2208 Rn, u \u2208 U) : Prob\u03be{(cid:107)B[(cid:98)x(y) \u2212 x](cid:107)\u221e \u2264 \u03b1 + S\u22121L1(Bx \u2212 [Bx]S)} \u2265 1 \u2212 \u0001.\n\nfor some \u03b1 > 0. Then there exist H = [h1, ..., hN ] \u2208 Rm\u00d7N and V = [V k(cid:96) \u2208 Rnk\u00d7n(cid:96)]K\nsatisfying\n\nof the form (11), namely,\n\n(12)\nk,(cid:96)=1,\n\nwith\n\n(c)\n\nand such that\n\nProb\n\n(cid:18)\n\n(a)\n\n(b)\n\n(cid:20)\n\nB = V B + H T A,\n(cid:107)V k(cid:96)(cid:107)\u221e,\u221e \u2264 2S\u22121 \u2200k, (cid:96) \u2264 K,\n\nu\u2208U uT hi + er\ufb01nv(\nmax\n\ner\ufb01nv(cid:0) \u0001\ner\ufb01nv(cid:0) \u0001\n(cid:19)\nu\u2208U uT hi + |(D\u03be)T hi| \u2264 \u03c1, 1 \u2264 i \u2264 N}\n\n)(cid:107)DT hi(cid:107)2\n\n\u2265 1 \u2212 \u0001.\n\n\u2264 2\u03b1\n\n\u0001\n2N\n\n(cid:21)\n\n2N\n\n2\n\n(cid:1)\n(cid:1) ,\n\n\u03c1 := max\n1\u2264i\u2264N\n\n\u039e+ := {\u03be : max\n\nThe latter exactly means that the exhibited H satis\ufb01es the condition Qs,\u221e(\u03ba) (see Proposition 3.1)\nfor s \u201cnearly as large as S,\u201d namely, s \u2264 \u03baS\n2 . Further, H = [h1, ..., hk], \u03c1 satisfy conditions (10)\n(and thus \u2013 condition B of Theorem 2.1), with \u03c1 being \u201cnearly \u03b1\u201d, namely, \u03c1 \u2264 2\u03b1\n. As a\nconsequence, under the premise of the proposition, we have for s \u2264 S\n8 (cf (11)):\n\ner\ufb01nv( \u0001\n2N )\ner\ufb01nv( \u0001\n2 )\n\nRisk\u221e(B(cid:98)xreg|s, D, \u03c5, \u0001) \u2264 8\u03b1\n\ner\ufb01nv( \u0001\n2N )\ner\ufb01nv( \u0001\n2 )\n\n+ 2s\u22121\u03c5.\n\n3.3 Condition Qs,\u221e(\u03ba), case r = 2: a veri\ufb01able suf\ufb01cient condition\nIn this section we consider the case of the representation structure B2 = (B, n1, ..., nK,(cid:107) \u00b7 (cid:107)2). A\nveri\ufb01able suf\ufb01cient condition for Qs,\u221e(\u03ba) is given by the following\nProposition 3.3 Let a sensing matrix A, a representation structure B2 be given. Let N = n1 + ... +\nnK, and let N \u00d7 N matrix V = [V k(cid:96)]K\nk,(cid:96)=1 (V k(cid:96) are nk \u00d7 n(cid:96)) and m \u00d7 N matrix H satisfy the\nrelation\n(13)\nB = V B + H T A.\n\nLet\n\n\u03bd\u2217(V ) = max\n1\u2264k,(cid:96)\u2264K\n\n\u03c3max(V k(cid:96)),\n\n(14)\n\nwhere \u03c3max stands for the maximal singular value. Then for all s \u2264 K we have:\n\nLs,\u221e(Bx) \u2264 L\u221e(H T Ax) + \u03bd\u2217(V )L1(Bx), \u2200x.\n\n(15)\nSuppose that the matrix A, the representation structure B2, the uncertainty set U, and the parameters\nD, \u0001 are given. Let us assume that the triple H, (cid:107) \u00b7 (cid:107) = L\u221e(\u00b7), and \u03c1 can be augmented by an\nappropriately chosen block N \u00d7 N matrix V to satisfy the system of convex constraints\n(13),\n(14). Our objective now is to synthesize the matrix H = [H k \u2208 Rm\u00d7nk ]K\nk=1 which satis\ufb01es the\nrelationship (3) with \u201cas good as possible\u201d value of \u03c1.\nLet us compute a bound on the probability of deviations of the variable (cid:107)(H k)T D\u03be(cid:107)2. Note that the\ndistribution of (cid:107)(H k)T D\u03be(cid:107)2\ni , where\n\u03b7, ..., \u03b7nk are i.i.d N (0, 1) and v[k] = [\u03c32\n[k]], \u03c3i[k] being the principal singular values\nof (H k)T D. To bound the deviation probabilities for \u03b6k we use the bound of [6] for the deviation of\nthe weighted \u03c72:\n\n2 coincides with that of the random variable \u03b6k =(cid:80)nk\n\n1[k], ..., \u03c32\nnk\n\nk=1 vi[k]\u03b72\n\nProb\n\nvi[k]\u03b72\n\ni \u2265 (cid:107)v[k](cid:107)1 +\n\n\u2264 2 exp\n\n\u2212\n\n\u03c4 2\n\n4(cid:107)v[k](cid:107)2\n\n2 + 4\u03c4(cid:107)v[k](cid:107)\u221e\n\n(cid:19)\n\n.\n\n(cid:40) nk(cid:88)\n\ni=1\n\n(cid:41)\n\n(cid:18)\n\n\u221a\n\n2(cid:107)v[k](cid:107)2\u03c4\n\n7\n\n\fWhen substituting (cid:107)v[k](cid:107)\u221e = \u03c32\n2, where\n\u03c3max[k] is the maximal singular value and (cid:107)\u03c3[k](cid:107)2 is the Frobenius norm of H T D, after a simple\nalgebra we come to\n\nnk, and (cid:107)v[k](cid:107)1 = (cid:107)\u03c3[k](cid:107)2\n\nmax[k]\n\nmax[k], (cid:107)v[k](cid:107)2 \u2264 \u03c32\n(cid:113)\n\n(cid:27)\n4 ln(2K\u0001\u22121) + 2(cid:112)nk ln(2K\u0001\u22121)\n\nProb\n\n(cid:107)(D\u03be)T H[k](cid:107)2 \u2265 (cid:107)\u03c3[k](cid:107)2 + \u03c3max[k]\n\n(cid:26)\n\n\u2264 \u0001\nK\n\n.\n\n\u221a\n\n(cid:20)\n\nLet \u00b5U (Hk) = max\n\nu\u2208U (cid:107)uT H[k](cid:107)2. Then the chance constraint\nProb(cid:8)\u03be : \u00b5U (H k) + (cid:107)(D\u03be)T H[k](cid:107)2 \u2264 \u03c1, 1 \u2264 k \u2264 K(cid:9) \u2265 1 \u2212 \u0001,\n(cid:21)\n(cid:113)\n4 ln(2K\u0001\u22121) + 2(cid:112)nk ln(2K\u0001\u22121)\n\nis satis\ufb01ed for\n\u03c1 \u2265 max\n(here (cid:107)\u00b7(cid:107)F stands for the Frobenius norm). In particular, in the case U = {0} (there is no nuisance),\nthe set Gs of admissible \u03ba, \u03c1 can be safely approximated by the computationally tractable convex\nset\n\n\u00b5U (H k) + (cid:107)DT H[k](cid:107)F + \u03c3max(DT H[k])\n\nk\n\n(cid:26)\n\nG\u2217\ns =\n\n(\u03ba, \u03c1) : \u2203H = [H k \u2208 Rm\u00d7nk ]K\n\nk=1, V = [V k(cid:96) \u2208 Rnk\u00d7n(cid:96)]K\n, 1 \u2264 k, (cid:96) \u2264 K\n\nk,(cid:96)=1\n\nB = BV + H T A, \u03c3max(V k(cid:96)) \u2264 \u03ba\ns\n\n4 ln(2K\u0001\u22121) + 2(cid:112)nk ln(2K\u0001\u22121), 1 \u2264 k \u2264 K\n\n(cid:113)\n\n(cid:27)\n\n.\n\n\u03c1 \u2265 (cid:107)DT H[k](cid:107)F + \u03c3max(DT H[k])\n\nWe have mentioned in Introduction that, to the best of our knowledge, the only previously proposed\nveri\ufb01able suf\ufb01cient condition for the validity of block (cid:96)1 recovery is the Mutual Block Incoherence\ncondition [9]. We aim now to demonstrate that this condition is covered by Proposition 3.3.\nThe Mutual Block Incoherence condition deals with the case where B = I and all block norms are\n(cid:107) \u00b7 (cid:107)2-norms. Let the sensing matrix A in question be partitioned as A = [A[1], ..., A[K]], where\nA[k], k = 1, ..., K, has nk columns. Let us de\ufb01ne the mutual block-incoherence \u00b5 of A w.r.t. the\nrepresentation structure in question as follows:\n\n(cid:0)C\u22121\nk AT [k]A[(cid:96)](cid:1) ,\n\n\u00b5 = max\nk(cid:54)=(cid:96)\n\n1\u2264k,(cid:96)\u2264K,\n\n\u03c3max\n\n[Ck := AT [k]A[k]]\n\n(16)\n\nprovided that all matrices Ck, 1 \u2264 k \u2264 K, are nonsingular, otherwise \u00b5 = \u221e. Note that in the case\nof the standard representation structure, the just de\ufb01ned quantity is nothing but the standard mutual\nincoherence known from the Compressed Sensing literature.\nWe have the following observation.\nProposition 3.4 Given m \u00d7 n sensing matrix A and a representation structure B2 with B = I,\n1 \u2264 k \u2264 K, let A = [A[1], ..., A[K]] be the corresponding partition of A.\nLet \u00b5 be the mutual block-incoherence of A with respect to B2. Assuming \u00b5 < \u221e, we set\n\n1\n\n[A[1]C\u22121\n\n1 , A[2]C\u22121\n\n2 , ..., A[K]C\u22121\n\nH =\n\n1 + \u00b5\n\n(17)\nThen the contrast matrix H along with the matrix V = I \u2212 H T A satis\ufb01es condition (13) (where\nB = I) and condition (14) with \u03bd\u2217(V ) \u2264 \u00b5s\n1+\u00b5 . As a result, applying Proposition 3.3, we conclude\nthat whenever\n\nK ], Ck = AT [k]A[k].\n\n2\u00b5\nthe pair (H, L\u221e(\u00b7)) satis\ufb01es Qs,\u221e(\u03ba) with \u03ba = \u00b5s\n1+\u00b5 < 1/2.\n\n1 + \u00b5\n\n,\n\ns <\n\n(18)\n\nNote that Proposition 3.4 essentially covers the results of [9] where the authors prove, under a\ncondition which is marginally stronger than that of\n(18), that an appropriate version of block-(cid:96)1\nrecovery allows to recover exactly every block-sparse signal from the noiseless observation y = Ax.\n\n8\n\n\fReferences\n[1] F. Bach. Consistency of the group lasso and multiple kernel learning. J. Mach. Learn. Res.,\n\n9:1179\u20131225, 2008.\n\n[2] Z. Ben-Haim and Y. C. Eldar. Near-oracle performance of greedy block-sparse estimation\ntechniques from noisy measurements. Technical report, 2010. http://arxiv.org/abs/\n1009.0906.\n\n[3] P. J. Bickel, Y. Ritov, and A. B. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector.\n\nAnnals of Stat., 37(4):1705\u20131732, 2008.\n\n[4] P. B\u00a8uhlmann and S. van de Geer. On the conditions used to prove oracle results for the Lasso.\n\nElectron. J. Statist., 3:1360\u20131392, 2009.\n\n[5] E. J. Cand`es and T. Tao. Decoding by linear programming. IEEE Trans. Inf. Theory, 51:4203\u2013\n\n4215, 2006.\n\n[6] L. Cavalier, G. K. Golubev, D. Picard, and A. B. Tsybakov. Oracle inequalities for inverse\n\nproblems. Ann. Statist., 30(3):843\u2013874, 2002.\n\n[7] A. Chambolle. An algorithm for total variation minimization and applications. Journal of\n\nMathematical Imaging and Vision, 20(1-2):89\u201397, 2004.\n\n[8] C. Chesneau and M. Hebiri. Some theoretical results on the grouped variables Lasso. Mathe-\n\nmatical Methods of Statistics, 27(4):317\u2013326, 2008.\n\n[9] Y. C. Eldar, P. Kuppinger, and H. B\u00a8olcskei. Block-sparse signals: Uncertainty relations and\n\nef\ufb01cient recovery. IEEE Trans. on Signal Processing, 58(6):3042\u20133054, 2010.\n\n[10] Y. C. Eldar and M. Mishali. Robust recovery of signals from a structured union of subspaces.\n\nIEEE Trans. Inf. Theory, 55(11):5302\u20135316, 2009.\n\n[11] J. Huang and T. Zhang. The bene\ufb01t of group sparsity. Annals of Stat., 38(4):1978\u20132004, 2010.\n[12] G. M. James, P. Radchenko, and J. Lv. Dasso: connections between the Dantzig selector and\n\nLasso. J. Roy. Statist. Soc. Ser. B, 71(1):127\u2013142, 2009.\n\n[13] A. B. Juditsky, F. K\u0131l\u0131nc\u00b8-Karzan, and A. S. Nemirovski. Veri\ufb01able conditions of (cid:96)1 recovery\nfor sparse signals with sign restrictions. Math. Progr., 127(1):89\u2013122, 2010. http://www.\noptimization-online.org/DB_HTML/2009/03/2272.html.\n\n[14] A. B. Juditsky and A. S. Nemirovski. Accuracy guarantees for (cid:96)1-recovery. Techni-\ncal report, 2010. http://www.optimization-online.org/DB_HTML/2010/10/\n2778.html.\n\n[15] A. B. Juditsky and A. S. Nemirovski. On veri\ufb01able suf\ufb01cient conditions for sparse signal\nrecovery via (cid:96)1 minimization. Math. Progr., 127(1):57\u201388, 2010. Special issue on machine\nlearning.\n\n[16] H. Liu and J. Zhang. Estimation consistency of the group Lasso and its applications. Journal\n\nof Machine Learning Research - Proceedings Track, 5:376\u2013383, 2009.\n\n[17] H. Liu, J. Zhang, X. Jiang, and J. Liu. The group Dantzig selector. Journal of Machine\n\nLearning Research - Proceedings Track, 9:461\u2013468, 2010.\n\n[18] K. Lounici, M. Pontil, A. Tsybakov, and S. van de Geer. Oracle inequalities and optimal\ninference under group sparsity. Technical report, 2010. http://arxiv.org/pdf/1007.\n1771.\n\n[19] Y. Nardi and A. Rinaldo. On the asymptotic properties of the group Lasso estimator for linear\n\nmodels. Electron. J. Statist., 2:605\u2013633, 2008.\n\n[20] G. Obozinski, M. J. Wainwright, and M. I. Jordan. Support union recovery in high-dimensional\n\nmultivariate regression. Annals of Stat., 39(1):1\u201347, 2011.\n\n[21] L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms.\n\nPhysica D, 60(1-4):259\u2013268, 1992.\n\n[22] M. Stojnic, F. Parvaresh, and B. Hassibi. On the reconstruction of block-sparse signals with an\noptimal number of measurements. IEEE Trans. on Signal Processing, 57(8):3075\u20133085, 2009.\n[23] M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. J.\n\nRoy. Stat. Soc. Ser. B, 68(1):49\u201367, 2006.\n\n9\n\n\f", "award": [], "sourceid": 734, "authors": [{"given_name": "Fatma", "family_name": "Karzan", "institution": null}, {"given_name": "Arkadi", "family_name": "Nemirovski", "institution": null}, {"given_name": "Boris", "family_name": "Polyak", "institution": null}, {"given_name": "Anatoli", "family_name": "Juditsky", "institution": null}]}